Method for sharing emotions through the creation of three-dimensional avatars and their interaction

ABSTRACT

A two-dimensional image is transformed into at least one portion of a human or animal body into a three-dimensional model. An image is acquired that includes the at least one portion of the human or animal body. An identification is made of the at least one portion within the image. Searches are made for features indicative of the at least one portion of the human or animal body within the at least one portion. One or more identifications are made of a set of landmarks corresponding to the features. An alignment is a deformable mask including the set of landmarks. The deformable mask includes a number of meshes corresponding to the at least one portion of the human or animal body. The 3D model is animated by dividing it into concentric rings, quasi rings and applying different degrees of rotation to each ring.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/853,488, filed Dec. 22, 2017, which is a continuation of U.S. patentapplication Ser. No. 14/456,759, filed Aug. 11, 2014, which is acontinuation-in-part of International Patent Application No.PCT/EP2013/052788, which has an international filing date of Feb. 12,2013 and claims priority to European Patent Application No. 12155243.4,filed Feb. 13, 2012, and which claims priority under 35 U.S.C. § 119 toU.S. Provisional Patent Application No. 62/018,766, filed Jun. 30, 2014.The present application incorporates herein by reference the disclosuresof each of the above-referenced applications in their entireties.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method for the sharing of emotionsthrough the creation of three-dimensional avatars and theirinteractions, and more particularly with the creation of such avatarsfor use with mobile devices, cloud based systems and the like.

Description of the Related Art

In the last few years, the development of powerful mobile devices andtheir diffusion worldwide has produced a global change in which peopleare moving away from using desktop computers to using a mobileequivalent, for example, mobile devices, laptops, notebooks and othersmall portable electronic devices which can connect to the web.

The use of such mobile devices has allowed the phenomenon called “socialnetworking” to become an important part in the day-to-day activities ofmany people's lives. Today, hundreds of millions of people use theirvirtual identities to communicate and interact with other people overthe web, internet and the like via social networking sites. In addition,these virtual identities are used to play games over the web, internetand the like.

While this widespread phenomenon provides the ability for contactbetween people any time of day or night, it is difficult to determineexactly how people are feeling by looking at their virtual identity. Itwould be a development of the “social networking” phenomenon to be ableto interact with people on social networking sites as if it were reallife, so that a person's moods, feelings, sentiments etc. can be madeavailable via the web, internet and the like.

This widespread phenomenon provides the ability for continuous contactbetween people any time of day or night; it is difficult to determineexactly how people are feeling by looking at their virtual identity. Itwould be a development of the “social networking” phenomenon to be ableto interact with people on social networking sites as if it were reallife, so that a person's moods, feelings, sentiments etc. can be madeavailable via the web, internet and the like.

In the last few years, the development of powerful mobile handsets andtheir diffusion is producing a new global change as most people aremoving from desktop to mobile devices. However, a person's virtualidentity is always with him/her as it becomes part of day-to-day life.

SUMMARY OF THE INVENTION

An object of the present invention to provide a method of creating athree-dimensional avatar.

Another object of the present invention is to provide a method ofcreating a three-dimensional avatar which maintains the morphology ofthe face.

Another object of the present invention is to provide a method ofcreating a three-dimensional avatar which provides a digitalrepresentation of feelings, sentiments, mood and the like,

A further object of the present invention is to provide a method ofcreating a three-dimensional avatar which provides a digitalrepresentation of feelings, sentiments, mood and the like wheninteracting with other people or animals using mobile devices, cloudsystems and other communication devices. . . .

It is another object of the present invention to provide methods forproviding representations of a digital identity of a person or animal.

A further object of the present invention to provide a representation ofa digital identity of a person which can be carried around and throughwhich it is always possible to understand feelings of people friendsand/or animals by simply looking at them as if it were real life, witheach digital identity being provided by a 3D avatar representingfeeling, sentiments, mood, and the like, of a user.

These and other objects of the present invention are achieved fortransforming a two-dimensional image of at least one portion of a humanor animal body into a three-dimensional model. An image is acquired thatincludes the at least one portion of the human or animal body. Anidentification is made of the at least one portion within the image.Searches are made for features indicative of the at least one portion ofthe human or animal body within the at least one portion. One or moreidentifications are made of a set of landmarks corresponding to thefeatures. An alignment is a deformable mask including the set oflandmarks. The deformable mask includes a number of meshes correspondingto the at least one portion of the human or animal body. The 3D model isanimated by dividing it into concentric rings, quasi rings and applyingdifferent degrees of rotation to each ring.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1(a)-(e) illustrates various embodiments of cloud infrastructuresthat can be utilized with the present invention;

FIG. 2 illustrates a “life cycle” of a “living portrait” in accordancewith one embodiment of the present invention.

FIG. 3 illustrates high functionality built into a “cloud client” inaccordance with one embodiment of the present invention.

FIG. 4 illustrates options for a user playing with his “living portrait”in accordance with one embodiment of the present invention.

FIG. 5 illustrates a “credit manager” subsystem of one embodiment of thepresent invention.

FIG. 6 illustrates a “get credits” subsystem in accordance with oneembodiment of the present invention.

FIG. 7 illustrates a social networking activity diagram in accordancewith one embodiment of the present invention.

FIG. 8 illustrates cross-social network integration in accordance withone embodiment of the present invention.

FIG. 9 illustrates the steps in creating a 3D face model in accordancewith one embodiment of the present invention.

FIG. 10 illustrates one embodiment of the steps in acquiring a sourceimage for creating the 3D face model in FIG. 9.

FIGS. 11 and 12 illustrate one embodiment of the recognition of an areaincluding a human or animal face.

FIGS. 13A, 13B and 13C illustrate one embodiment of the process offitting a model to a face inside an image.

FIG. 14 illustrates one embodiment of a grid comprising a plurality oftriangles that is applied to a face.

FIG. 15 illustrates one embodiment of the steps in creating a modeltexture.

FIG. 16 illustrates one embodiment of steps that can be used in anactivity diagram with the present invention.

FIG. 17 illustrates 3DoM general architecture in accordance with thepresent invention.

FIG. 18 illustrates one embodiment of 3DoM logical architecture.

FIGS. 19 to 21 illustrate certain embodiments of applications of anavatar created in accordance with one embodiment of the presentinvention.

FIGS. 22-24 illustrate one embodiment of a mobile device with an appthat can be used with the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As used herein, the term engine refers to software, firmware, hardware,or other component that can be used to effectuate a purpose. The enginewill typically include software instructions that are stored innon-volatile memory (also referred to as secondary memory) and aprocessor with instructions to execute the software. When the softwareinstructions are executed, at least a subset of the softwareinstructions can be loaded into memory (also referred to as primarymemory) by a processor. The processor then executes the softwareinstructions in memory. The processor may be a shared processor, adedicated processor, or a combination of shared or dedicated processors.A typical program will include calls to hardware components (such as 1/0devices), which typically requires the execution of drivers. The driversmay or may not be considered part of the engine, but the distinction isnot critical.

As used herein, the term database is used broadly to include any knownor convenient means for storing data, whether centralized ordistributed, relational or otherwise.

As used herein a mobile device includes, but is not limited to, a cellphone, such as Apple's iPhone®, other portable electronic devices, suchas Apple's iPod Touches®, Apple's iPads®, and mobile devices based onGoogle's Android® operating system, and any other portable electronicdevice that includes software, firmware, hardware, or a combinationthereof that is capable of at least receiving a wireless signal,decoding if needed, and exchanging information with a server. Typicalcomponents of mobile device may include but are not limited topersistent memories like flash ROM, random access memory like SRAM, acamera, a battery, LCD driver, a display, a cellular antenna, a speaker,a BLUETOOTH® circuit, and WIFI circuitry, where the persistent memorymay contain programs, applications, and/or an operating system for themobile device.

As used herein, the term “computer” is a general purpose device that canbe programmed to carry out a finite set of arithmetic or logicaloperations. Since a sequence of operations can be readily changed, thecomputer can solve more than one kind of problem. A computer can includeof at least one processing element, typically a central processing unit(CPU) and some form of memory. The processing element carries outarithmetic and logic operations, and a sequencing and control unit thatcan change the order of operations based on stored information.Peripheral devices allow information to be retrieved from an externalsource, and the result of operations saved and retrieved. Computer alsoincludes a graphic display medium.

As used herein, the term “internet” is a global system of interconnectedcomputer networks that use the standard web protocol suite (TCP/IP) toserve billions of users worldwide. It is a network of networks thatconsists of millions of private, public, academic, business, andgovernment networks, of local to global scope, that are linked by abroad array of electronic, wireless and optical networking technologies.The internet carries an extensive range of information resources andservices, such as the inter-linked hypertext documents of the World WideWeb (WWW) and the infrastructure to support email. The communicationsinfrastructure of the internet consists of its hardware components and asystem of software layers that control various aspects of thearchitecture.

As used herein, the term “extranet” is a computer network that allowscontrolled access from the outside. An extranet can be an extension ofan organization's intranet that is extended to users outside theorganization in isolation from all other internet users. An extranet canbe an intranet mapped onto the public internet or some othertransmission system not accessible to the general public, but managed bymore than one company's administrator(s). Examples of extranet-stylenetworks include but are not limited to:

-   -   LANs or WANs belonging to multiple organizations and        interconnected and accessed using remote dial-up    -   LANs or WANs belonging to multiple organizations and        interconnected and accessed using dedicated lines    -   Virtual private network (VPN) that is comprised of LANs or WANs        belonging to multiple organizations, and that extends usage to        remote users using special “tunneling” software that creates a        secure, usually encrypted network connection over public lines,        sometimes via an ISP.

As used herein, the term “Intranet” is a network that is owned by asingle organization that controls its security policies and networkmanagement. Examples of intranets include but are not limited to:

-   -   ALAN    -   A Wide-area network (WAN) that is comprised of a LAN that        extends usage to remote employees with dial-up access    -   A WAN that is comprised of interconnected LANs using dedicated        communication lines    -   A Virtual private network (VPN) that is comprised of a LAN or        WAN that extends usage to remote employees or networks using        special “tunneling” software that creates a secure, usually        encrypted connection over public lines, sometimes via an        Internet Service Provider (ISP).

For purposes of the present invention, the Internet, extranets andintranets collectively are referred to as (“Network Systems”).

For purposes of the present invention, the term “Infrastructure” means,information technology, the physical hardware used to interconnectcomputers and users, transmission media, including telephone lines,cable television lines, and satellites and antennas, and also therouters, aggregators, repeaters, computers, network devices,applications, and other devices that control transmission paths,software used to send, receive, and manage the signals that aretransmitted, and everything, both hardware and software, that supportsthe flow and processing of information.

As used herein, “event message” or “event” is defined as a change instate. An event is anything that happens, or is contemplated ashappening in message form or event form relating to infrastructure. Anevent can include a time stamp, and a name for the entity changingstate.

As used herein “Living Portrait®” refers to a 3D deformable face modelbased on a photo of a human or animal face. “Living Portrait®” is atrademark of Mach 3D granted by US PTO (Registered May 21, 2013, U.S.Pat. No. 4,338,828 under Int. Cl. 9).

Java® is a known programming language developed by Sun Microsystems, asubsidiary of Oracle Corporation. Much of its syntax is derived from Cand C++ but has a simpler object model and fewer low-level facilities.

As used herein “C” is a general purpose computer programming languagethat is widely used for developing portable application software.

As used herein “C++” is a popular programming language used for systemssoftware, application software, device drivers, embedded software, highperformance server and client application as well as entertainmentsoftware.

As used herein “OpenCV” refers to Open Source Computer Vision, a libraryof programming functions for real time computer vision.

As used herein “Cloud or Mobile Device Computing” refers to the deliveryof computing as a service rather than a product. Shared resources andinformation are provided to computers and other devices as a utilityover a network, typically the Network Systems.

As used herein “C or Mobile Device client” refers to computer hardwareand/or software that rely on “cloud computing” for application delivery.This includes mobile devices, tablets, other mobile devices, desktopsand Network System applications that are connectable to the Cloud orMobile device for Cloud or Mobile Device Computing. It includes anyprogram that can interact with the platform, and in particular, the 3DoMPlatform described below, which is hosted in the Cloud.

As used herein “Cloud or Mobile Device Application” refers to cloud ormobile device application services or “software as a service” (SaaS)which deliver software over the Network Systems eliminating the need toinstall and run the application on a device.

As used herein “Cloud or Mobile Device Platform” refers to a cloud ormobile device platform services or “platform as a service” (PaaS) whichdeliver a computing platform and/or solution stack as a service, andfacilitates the deployment of applications without the cost andcomplexity of obtaining and managing the underlying hardware andsoftware layers.

As used herein “Cloud Infrastructure” refers to cloud infrastructureservices or “infrastructure as a service” (IaaS) which deliver computerinfrastructure as a service with raw block storage and networking.

As used herein “Server” refers to server layers that consist of computerhardware and/or software products specifically designed for the deliveryof cloud or mobile device services.

“Image histogram” refers to a graphical representation of tonaldistribution in a digital image. It plots the number of pixels for eachtonal value, and is used to judge and change tonal distribution of anentire image at a glance.

As used herein “App” refers to application, for example, Network System,cloud and mobile device applications.

As used herein “3me®” or “MoodEe®” is a virtual object that can be sentto the Living Portrait of a recipient to catch his/her attention. It canbe considered to be like saying “hello” or winking. “3me®” and “MoodEe®”are trademarks of Mach-3D. 3me® or MoodEe® will be described in moredetail below.

As used herein “3DoM® Platform” refers to a particular Cloud or MobileDevice Platform that is operated by Mach-3D and through which a user canbe linked to his/her social networking site. “3DoM®” is a trademark ofMach-3D.

As used herein “MoodMe®” refers to the technologies described in thepresent Patent filing. “MoodMe®” is a trademark of Mach-3D granted by USPTO (Registered May 14, 2013, U.S. Pat. No. 4,334,062 under Int. Cl. 9).

As used herein “Vertex” means: (i) a point, including but not limited toas of an angle, polygon or mesh structure (which can be a primitive),polyhedron, graph, network and the like that terminates a line or curve;(ii) the intersection of two or more lines or curves and (iii) a pointwhere an axis of an ellipse, parabola, or hyperbola intersects the curveitself.

As used herein “Mesh” means a collection of vertexes—points in 3Dspace—that can be connected in different ways. The shapes that areformed by connecting these vertexes are called primitives. Theprimitives are building blocks that are [it together to form a mesh. Notall of the vertexes need to be connected and points can be drawn as theprimitives. Pairs of vertexes can be connected as the primitives. Threeor more vertexes can be drawn as the primitives.

As used herein “Living Portrait” (LP) is a 3D model of a human (oranimal) face generated automatically from one single picture. This modelcan be animated with a simulation of life, hence the term Living, and isa realistic representation of the human face, hence the term Portrait.

In one embodiment the present invention provides methods of creating athree-dimensional avatar.

In one embodiment the present invention is to provide a method ofcreating a three-dimensional avatar which maintains the morphology ofthe face.

In one embodiment the present invention is provides methods of creatinga three-dimensional avatar which provides a digital representation offeelings, sentiments, mood and the like,

In one embodiment the present invention provides methods of creating athree-dimensional avatar. The three-dimensional avatar provides adigital representation of feelings, sentiments, mood and the like wheninteracting with other people or animals using mobile devices, cloudsystems and other communication devices.

In one embodiment the present invention provides methods for creatingrepresentations of a digital identity of a person or animal.

In one embodiment the present invention provides a representation of adigital identity of a person which can be carried around and throughwhich it is always possible to understand feelings of people friendsand/or animals by simply looking at them as if it were real life. Eachdigital identity is provided by a 3D avatar representing feeling,sentiments, mood, and the like, of a user.

In one embodiment a method transforms a two-dimensional image of atleast one portion of a human or animal body into a three-dimensionalmodel. A search is made of features indicative of at least a portion ofthe human or animal body within the at least one portion. A set oflandmarks is identified that corresponds to the features. A 3Ddeformable mask is aligned, including the set of landmarks to create a3D model of the face respecting its morphology, the deformable maskincluding a number of mesh shapes that correspond to at least oneportion of the human or animal. The 3D model is animated by dividing itinto concentric rings and applying different degrees of rotation to eachring.

In one embodiment a cloud structure is utilized. As a non-limitingexample, one embodiment of a cloud structure is illustrated in FIGS.1(a)-1(e),

The cloud based system 110 includes a third party service provider 120,that is provided by the methods used with the present invention, thatcan concurrently service requests from several clients without userperception of degraded computing performance as compared to conventionaltechniques where computational tasks can be performed upon a client or aserver within a proprietary intranet. The third party service provider120 (e.g., “cloud”) supports a collection of hardware and/or softwareresources 122. The hardware and/or software resources 122 can bemaintained by an off-premises party, and the resources 122 can beaccessed and utilized by identified users over Network Systems.Resources 122 provided by the third party service provider 120 can becentrally located and/or distributed at various geographic locations.For example, the third party service provider 120 can include any numberof data center machines that provide resources 122. The data centermachines can be utilized for storing/retrieving data, effectuatingcomputational tasks, rendering graphical outputs, routing data, and soforth.

In one embodiment, the third party service provider 120 can provide anynumber of resources 122 such as data storage services, computationalservices, word processing services, electronic mail services,presentation services, spreadsheet services, gaming services, websyndication services (e.g., subscribing to a RSS feed), and any otherservices or applications that are conventionally associated withpersonal computers and/or local servers. Further, utilization of anynumber of third party service providers similar to the third partyservice provider 120 is contemplated. According to an illustration,disparate third party service providers can be maintained by differingoff-premise parties and a user can employ, concurrently, at differenttimes, and the like, all or a subset of the third party serviceproviders.

By leveraging resources 122 supported by the third party serviceprovider 120, limitations commonly encountered with respect to hardwareassociated with clients and servers within proprietary intranets can bemitigated. Off-premises parties, instead of users of clients or networkadministrators of servers within proprietary intranets, can maintain,troubleshoot, replace and update the hardware resources 122. Further,for example, lengthy downtimes can be mitigated by the third partyservice provider 120 utilizing redundant resources 122; thus, if asubset of the resources 122 are being updated or replaced, the remainderof the resources 122 can be utilized to service requests from users.According to this example, the resources 122 can be modular in nature,and thus, resources 122 can be added, removed, tested, modified, etc.while the remainder of the resources 122 can support servicing userrequests. Moreover, hardware resources 122 supported by the third partyservice provider 120 can encounter fewer constraints with respect tostorage, processing power, security, bandwidth, redundancy, graphicaldisplay rendering capabilities, etc. as compared to conventionalhardware associated with clients and servers within proprietaryintranets.

The cloud based system 110 can include a client device 124 that employsresources 122 of the third party service provider 120. Although oneclient device 124 is depicted, it is to be appreciated that the cloudbased system 100 can include any number of client devices similar to theclient device 124, and the plurality of client devices can concurrentlyutilize supported resources 122. By way of illustration, the clientdevice 124 can be a desktop device (e.g., personal computer), mobiledevice, and the like. Further, the client device 124 can be an embeddedsystem that can be physically limited, and hence, it can be beneficialto leverage resources 122 of the third party service provider 120.

Resources 122 can be shared amongst a plurality of client devicessubscribing to the third party service provider 120. According to anillustration, one of the resources 122 can be at least one centralprocessing unit (CPU), where CPU cycles can be employed to effectuatecomputational tasks requested by the client device 124. Pursuant to thisillustration, the client device 124 can be allocated a subset of anoverall total number of CPU cycles, while the remainder of the CPUcycles can be allocated to disparate client device(s). Additionally oralternatively, the subset of the overall total number of CPU cyclesallocated to the client device 124 can vary over time. Further, a numberof CPU cycles can be purchased by the user of the client device 124. Inaccordance with another example, the resources 122 can include datastore(s) that can be employed by the client device 124 to retain data.The user employing the client device 124 can have access to a portion ofthe data store(s) supported by the third party service provider 120,while access can be denied to remaining portions of the data store(s)(e.g., the data store(s) can selectively mask memory based uponuser/device identity, permissions, and the like). It is contemplatedthat any additional types of resources 122 can likewise be shared.

The third party service provider 120 can further include an interfacecomponent 128 that can receive input(s) from the client device 124and/or enable transferring a response to such input(s) to the clientdevice 124 (as well as perform similar communications with any disparateclient devices). According to an example, the input(s) can berequest(s), data, executable program(s), etc. For instance, request(s)from the client device 124 can relate to effectuating a computationaltask, storing/retrieving data, rendering a user interface, and the likevia employing one or more resources 122. Further, the interfacecomponent 128 can obtain and/or transmit data over a network connection.According to an illustration, executable code can be received and/orsent by the interface component 128 over the network connection.Pursuant to another example, a user (e.g. employing the client device124) can issue commands via the interface component 128.

Moreover, the third party service provider 120 includes a dynamicallocation component 110 that apportions resources 122 (e.g., hardwareresource(s)) supported by the third party service provider 120 toprocess and respond to the input(s) (e.g., request(s), data, executableprogram(s), . . . ) obtained from the client device 124.

Although the interface component 128 is depicted as being separate fromthe dynamic allocation component 110, it is contemplated that thedynamic allocation component 110 can include the interface component 128or a portion thereof. The interface component 128 can provide variousadaptors, connectors, channels, communication paths, etc. to enableinteraction with the dynamic allocation component 110.

In one embodiment of the present invention, a structure is placed arounda geometric outline of the face. As a non-limiting example, thegeometric figure can be an oval, a quasi-oval or a geometry that closelyfollows the outline of the face. As a non-limiting example, in oneembodiment the geometric outline of the face is based on Phi and GoldenRatio proportions. In one embodiment the structure is two shapes thatcan be concentric, around the geometric outline of the face. In oneembodiment, two rings are used because the realism of the human (oranimal) face is deemed satisfactory by a group of observers whereasusing 1 ring only was not considered realistic enough. Meanwhile, trialsof human (or animal) faces with 3 rings made the model, in particularthe model rotation, computationally more complex and the 3D animationsbecame heavy without adding significant human—like realism. As anon-limiting example, 2 rings are considered to be the best trade-offbetween realism and resources requirement. In one embodiment, the ringssurround the geometric outline of the face and rotate with lower speedand smaller angles than the inner geometric outline of the face tocreate a realistic movement of the face. In one embodiment, additionalface information is obtained by extending the geometric outline of theface. Each ring moves with a different angular speed relative to thegeometric outline of the face. The angles have been calculated,empirically, and are described as follows.

Not all the vertices in an LP are moved according to the defined rules.To make the LP animation more realistic, the LP general model (LPGM) hasbeen structured to cover not only the geometric outline of the face butalso an area containing the image background. From the animation pointof view, the LPGM is divided in many areas which are concentric rings;

-   -   Contour 3, the outer edge, far from the head geometric outline        of the face, including large portions of the background;    -   Contour 2, the area between Contour 3 and the face geometric        outline of the face, including hair and small portions of the        background;    -   Upper Contour, an upper portion of Contour 2, including all        points on the upper side of the eyebrow; and    -   Lower Contour, a lower portion of Contour 2, including all        points on the lower side of the chin.

To evaluate X-, Y- and Z-axis rotation to be applied to each vertex ofthe LP, the following rules are applied:

(1) If a vertex belongs to Contour 3, no rotation is applied therebyanchoring all points in Contour 3 to the image background;

(2) If a vertex belongs to Lower Contour, X- and Y-rotations are reducedby 50% of their value with the Z-axis rotation being reduced to 10% ofthe original value;

(3) If a vertex belongs to Upper Contour, X-, Y- and Z-axis rotationsare reduced to 60% of their original value;

(4) if a vertex belongs to Contour 2, X-, Y- and Z-axis rotations arereduced to 30% of their original value; and

(5) All other vertices are moved according to the defined rules.

Note: These values are given as example only and can be modified indifferent implementations. The principle is to have different level ofrotation for areas around the face in order to generate a more realisticanimation of the head rotation.

This kind of movement, especially between Contour 2 and Contour 3,introduces very noticeable deformations. Because Contour 3 is fixed, allthe textures are stretched with a disturbing effect. To reduce thiseffect, a gradient is applied, which gradually makes textures completelyopaque in the area next to the center of the LP, and almost totallytransparent in the area next to the top of Contour 3.

The approach used in this embodiment is empirical. The background doesnot rotate when the face rotates while the face inside the geometricoutline rotates at the full angle. The concentric rings rotate at lowerspeed (smaller angles) than the face inside the geometric outline tosimulate a more realistic rotation.

In this embodiment, it is taken into consideration that the lateralparts of the face (the sides of the head) are unknown because work isdone on the basis of only one front-facing picture. Therefore, theselateral parts can be created to show them when the face rotates.

A rotation of the geometric outline of the face and the rings occursaround a pivot point. In one embodiment, the pivot point has beencalculated, empirically, as follows, by an iterative process.

The origin of the three axes of rotation, the so-called pivot point, isobtained using the X- and Y-coordinates of the sixty-sixth landmark ofthe model LPGM. The Z value is modified to different values, fractionsof the depth of the model, such as 5/10, 6/10, . . . 15/10 . . . while,for each such value, a group of people gave their personal assessment ofwhat rotations gave the model the most realistic aspect when rotating.At the end of this iterative process, it was agreed that the Z value ofthe required point would be set equal to the 11/10 of the depth of themodel. In other embodiment, one could have a pivot point where the Zvalue is, for example, 12/10 of the depth of the model, or 10/10 (i.e. 1time) the depth.

It is to be noted that the geometric outline of the face need not be aperfect geometric outline of the face. Instead it is a geometricconfiguration that is defined by the actual face and can be elliptical,quasi elliptical, a non-perfect geometric outline of the face, asubstantially geometric outline of the face geometry with some linearsections and with different arch's.

In one embodiment the rings are obtained through a non-mathematicalprocess. It can be an iterative empirical process where many peoplewatch “living Portraits” (LP) moving, and then try to understand if theLP was not human like (i.e., realistic) enough and what could be done tomake it more human. Suggestions are retained after trial and error tointroduce rings. These are then implemented. People are grouped again towatch, comment, and so on for several months until a solution is foundwhich is the most human and then utilized. The approach used in thisembodiment is to put human's center stage rather than technology byasking real people how they felt, how they liked the different livingPortraits generated and which one was satisfactory, causing positiveemotional reactions and emotional engagement.

In one embodiment the geometric outline of the face is not a perfect,symmetrical geometric outline of the face but is determined by theapplication of ASM on the two-dimensional picture. The output is aseries of points where an ASM was applied to identify, as a non-limitingexample, about 60 landmarks). In one embodiment 96 landmarks areidentified and utilized.

In other embodiments, 2 additional points can be included to define theMouth. In one embodiment 3, 4, even 10 or 20 points and the like can beadded to the mouth, the nose, the eye, the eyebrows and the geometricoutline of the face.

It has been defined that the quality of the model with such 96 landmarkswas considered sufficient by a group of people and that a higher numberof landmarks would not automatically mean a better model as landmarksdepend on face characteristics and such face characteristics can berepresented realistically enough with such a number of landmarks.

As a non-limiting example, a training set of the ASM is used withthousands of pictures of people faces where all the landmarks/points areplaced also did not have perfect geometric outlines of the face. It hadpoints corresponding to the landmarks of each of these thousands ofpeople. The points are where they should be to respect the morphology ofthe people, not to attempt to be a perfect geometrical shape, includingbut not limited to an oval, quasi oval, ellipse, quasi ellipse and thelike.

The rings are not perfect shapes either. The “rings” are called Contour.Contour 2 (inner ring) and Contour 3 (outer ring) are the rings aroundthe “geometric outline of the face”. They are the areas delimited by asequence of straight lines that are themselves determined by trianglesand other geometries.

In another embodiment a method transforms a two-dimensional image of atleast one portion of a human or animal body into a three-dimensionalmodel. An image is acquired that includes the at least one portion ofthe human or animal body. An identification is made of the at least oneportion within the image. Searches are made for features indicative ofthe at least one portion of the human or animal body within the at leastone portion. One or more identifications are made of a set of landmarkscorresponding to the features. An alignment is a deformable maskincluding the set of landmarks. The deformable mask includes a number ofmeshes corresponding to the at least one portion of the human or animalbody. The 3D model is animated by dividing it into concentric rings andapplying different degrees of rotation to each ring.

In another embodiment a method is provided for transforming atwo-dimensional image, of at least one portion of a human or animalbody, into a three-dimensional model. An image is acquired that includesat least one portion of the human or animal body. The at least oneportion within the image is identified. A search is performed forfeatures indicative of the at least one portion of the human or animalbody within the at least one portion. Identification is made for a setof landmarks corresponding to the features. An alignment is made of adeformable mask including the set of landmarks, the deformable maskcomprising a number of polygons that correspond to the at least oneportion of the human or animal body. An animation is made of the 3Dmodel by dividing it into concentric rings and applying differentdegrees of rotation to each ring. In another embodiment, a method isprovided for transforming a two-dimensional image of at least oneportion of a human or animal body into a three-dimensional model. Anacquisition is made of an image including the at least one portion ofthe human or animal body. An identification is made of the at least oneportion within the image. A search is performed for features indicativeof the at least one portion of the human or animal body within the atleast one portion, A set of landmarks is identified that correspond tothe features. An alignment is performed of a deformable mask includingthe set of landmarks, the deformable mask including a number of meshesthat correspond to the at least one portion of the human or animal body.A texture is applied to each parameter forming the mask to provide thethree-dimensional.

In one embodiment the present invention allows people to communicatetheir feelings and mood through the Network Systems using their3-dimensional (3D) model created using a portrait of a human or animalface. As a non-limiting example, in one embodiment, the presentinvention does not provide an alternative to social networks ornetworking sites, but works with them, and, with other communicationtools, including but not limited to, Skype®, MSN®, Yahoo!®, and thelike, using Network Systems via a 3DoM® Platform connectable to socialnetworks or networking sites. Skype® and MSN® are trademarks ofMicrosoft Corporation, and Yahoo!® is a trademark of Yahoo! Inc.

In one embodiment, the methods use the cloud or mobile devices and canbe divided in three components: cloud or mobile device server, cloud ormobile device client, and cloud or mobile device applications as definedabove. For example, the cloud or mobile device server includes the 3DoM®platform, the cloud or mobile device client includes any mobile device,desktop and Network System application connected to the 3DoM® platform;and cloud or mobile device Applications include the applicationsprovided by the 3DoM® Platform.

As described herein, the creation of a LP is an element to the 3DoM®Platform, and requires the ability to transform a two-dimensional (2D)human or animal portrait into a 3D deformable model.

In one embodiment, there is a functionally of sharing LPs, moods andemotions through the structure as illustrated in FIGS. 1(a)-1(e). Aclient device, mobile device disclosed in FIGS. 22-24, or Network Systemshares a LP or moods with a third party service provider 120, FIG. 1(b),that allocates a resource 122, including but not limited to a CPU andmemory, to process the request and to store the LP and associatedinformation into a Data Store 146, FIG. 1(e). The mobile device can beused in every instance where the cloud is used, with correspondingcomponents as disclosed in FIGS. 22-24.

In one embodiment network usage needs to be minimized as well as theamount of data stored on the 3DoM® cloud or mobile device server. The3DoM® Platform has typical cloud or mobile device architecture: all thecomponents communicate exchanging messages that represent the way toimplements computing and application as a service. Data is sharedthrough the cloud or mobile device using the Network Systems as thecommunication channel utilizing the client device 124, the cloud ormobile device CPU and memory to process the associated information inthe data store.

A client device 124, FIG. 1(b), exchanges messages with the InterfaceComponent 126, FIG. 1(b), according to communication protocols with athird party service provider 120, FIG. 1(b), that allocates a resource122, FIG. 1(b), including but not limited to a CPU and memory, toprocess the message, to acknowledge it, respond to it and to store a logof this message into a Data Store 146, FIG. 1(e).

Changes of facial expressions, emotions, moods and accessories of the LPmay generate events that the LP publishes through an API (ApplicationProgramming Interface). Such events can be detected by otherapplications. This capability allows for the creation of animatedscenarios composed of one or more objects, either in 2D or 3D, in motioncombined with changing moods and accessories of a LP. In one example,the eyes of the LP are capable of following a moving object close to itsface. In another, an LP will wear a hat from a given brand and display aspecific emotion when a branded product appears.

In one embodiment, an ASM (Active Shape Model) algorithm is used. TheASM algorithm is based on statistical models of shapes of objects, toidentify the landmarks on the image.

FIG. 12 illustrates one embodiment of the execution of the ASMalgorithm.

In one embodiment the ASM algorithm adds a process of searching the mostsimilar image to the target object from the training sample set, anduses the shape model of the similar image instead of the average shapemodel to approximately express target object model. Now, searching themost similar image process will be introduced in detail.

One embodiment of a suitable ASM is as follows:

Assuming that there is a sample O of training images (the training set)composed of N training image samples, marking m feature points manually,the training sample set and the face image can be expressed by a shapevector [2]:

Ω={s ₁ , . . . ,s _(i) , . . . ,s _(N)}  (1)

s _(i)=(x _(i1) ,y _(i1) , . . . ,x _(ij) ,y _(ij) , . . . x _(im) ,y_(im))^(T)  (2)

where Si means the i-th training sample, (xii, Yij) is the coordinate ofthe j-th feature point of the i-th training sample.

Aligning the training set: After obtaining the training sample set 0, itis found that the distribution of the feature points is very confused,and it absolutely has no face shape, but the fuzzy face outline can beseen. Therefore the training sets need to be aligned. In one embodiment,a Procrustes algorithm is used to minimize the sum of the distances fromthe all shapes to the average shape.

As a non-limiting example, assume that M(Sc, S)[s] is a transformationto s with the rotation angle 8 for the rotational change and the Sc forthe scale. Therefore:

$\begin{matrix}{{{M( {{Sc},\theta} )}\begin{bmatrix}x_{ij} \\y_{ij}\end{bmatrix}} = \begin{pmatrix}{{{Sc}*\cos \; \theta*x_{ij}} - {{Sc}*\sin \; \theta*y_{ij}}} \\{{{Sc}*\sin \; \theta*x_{ij}} + {{Sc}*\cos \; \theta*y_{ij}}}\end{pmatrix}} & (3)\end{matrix}$

The aligning process from s₂ to s₁ is a process to solve the rotationangle 0, scale Sc and the displacement (tx,ty). The solution to theseparameters is given as the following equations.

$\mspace{79mu} \lbrack {\begin{matrix}X_{2} & {- Y_{2}} & W & O & { \prime \} ( \text{?} } \\Y_{2} & X_{2} & W & {()} & {{{S} < {{\prime\prime\prime}!} > {in}}\&} \\Z & 0 & {{x\_},} & & {,,} \\0 & Z & {- \frac{1}{2}} & {{.Y.d}\backslash} & {t,}\end{matrix} = ( {\begin{matrix}X_{i} \\Y_{i} \\C_{1} \\C_{2}\end{matrix}\text{?}\text{indicates text missing or illegible when filed}} } $

where:

$\begin{matrix}{\mspace{79mu} {X_{i} = {\sum\limits_{k = 1}^{n}{w_{k}x_{ik}}}}} & (5) \\{\mspace{79mu} {Y_{i} = {\sum\limits_{k = 1}^{n}{w_{k}y_{ik}}}}} & (6) \\ \mspace{79mu} {{{z = {{{\text{?}w\text{,}\text{\_}( {{.x};{k +}} \}} -} <}};},.} ) & (7) \\{\mspace{79mu} {W = {\sum\limits_{k = 1}^{n}w_{k}}}} & (8) \\{\mspace{79mu} {C_{1} = {\sum\limits_{k = 1}^{n}{w_{1}( {{x_{1k}x_{2\; k}} + {y_{1\; k}y_{2k}}} )}}}} & (9) \\{\mspace{79mu} {C_{2} = {\sum\limits_{k = 1}^{n}{w_{k}( {{y_{1k}x_{2k}} - {x_{1k}y_{2k}}} )}}}} & (10) \\{\text{?}\text{indicates text missing or illegible when filed}} & \;\end{matrix}$

After getting the rotation angle

, scale Sc and the displacement(tx,ty), then the training sets can beupdated by

s ₁ =M(Sc.O)[s.]+(t . . . ,t ₃ ,t _(y), . . . •(•

_

)  (11)

where i=1, 2, . . . N.

The principal component analysis: After alignment to the face images inthe training set, the statistical rule for the shape change can be foundby using the PCA (Principal Component Analysis) method.

Calculate the average shape:

$\begin{matrix}{\overset{\_}{s} = {\frac{1}{N}{\sum\limits_{i = 1}^{N}{\overset{\sim}{s}}_{i}}}} &  02 )\end{matrix}$

Calculate the covariance matrix:

$\begin{matrix}{\mspace{79mu} {{Cov} = {\frac{1}{{1V} - 1}{LN}\text{?}}}} & ( B \} \\{\text{?}\text{indicates text missing or illegible when filed}} & \;\end{matrix}$

Find out the eigenvalue .1 and the eigenvector p of the Cov, choose thelargest r eigenvalues [.11, .12, . . •.1r] such that.

$\begin{matrix}{\mspace{79mu} {\frac{\begin{matrix}{\prime O''} \\{{{.\pounds}\mspace{14mu} \ldots \mspace{14mu} 1},{{- o} - {1l}},\text{?}}\end{matrix}}{{{\text{?}''} - A};}{\text{?}\text{indicates text missing or illegible when filed}}}} & ( {\text{!}4} )\end{matrix}$

Choose the eigenvectors P=[p₁, P2, . . . ,Pr] corresponding to thelargest r eigenvalues, the statistical model of the training set can beobtained by:

$\begin{matrix}{\mspace{79mu} {s\overset{\text{?}}{= {s + {Pb}}}}} & (15) \\{\text{?}\text{indicates text missing or illegible when filed}} & \;\end{matrix}$

Gray matching and Color Matching: The average shape model approximatelyexpresses the target object, the optimal matching points are obtained bycalculating Markovian, and then the various parameters can be updated by

db::::pTdX  (16)

where dx is the displacement, db is the change of shape parameter.

In one embodiment, the former z points, whose gray value is the mostsimilar to the i-th feature point of the j-th image in the area withthis feature point as the center and the distance being d. Take d=2, z=9for example.

where the red point is the i-th feature point of the j-th image. Thepositions of the nine points marked above are just assumed points.

Calculate the mean gray value of these z points, marked as dgij. We knowthere are m feature points in each image, the gray information set dgjof the j-th image could be regained.

dgj=[dg ₁₁ , . . . dgij, . . . dgmY  (17)

where jE{1,z, . . . ,N}}, is the number of the training images.

Get the gray value for the feature points of each image projecting inthe target image, so the gray matrix could be gained.

G=[gi, . . . ,g ₁ , . . . g,]  (18)

where gi is the gray vector of the characteristic points of the j-thtraining image projecting in the target image.

If the target image shape model is very close to some image shape modelin the training set, the Euclidean distance will be small. In order tofind the most similar image to the target image, it is only needed tosolve the following problem.

$\begin{matrix}{i_{0} = {\arg\limits_{i}\mspace{14mu} {\min ( {( {{dgJ} - {gj}} )^{\prime}( {{d\; {g \cdot i}} - g} )} )}}} & (19)\end{matrix}$

The i₀-th image gotten in the formula (19) is the most similar image. Inone embodiment the i₀-th image is taken as the approximate expression tothe target image.

As anon-limiting example the ASM model used is determined from a numberof training sets, and the number of landmarks chosen is in accordancewith the processing that is required to create the 3D model or avatarform a 2D photo. As a non-limiting example, starting from a 2D photo, 93landmarks identifying face characteristics are utilized in an imageinterpretation phase. It will be appreciated the 93 landmarks are onlyone number of suitable landmarks and that other landmark numbers can beused. These landmarks are special points placed by ASM in correspondenceto a typical characteristic and will be described in more detail below.For purposes of the present invention a set of these landmarks is calledan ASHAPE. At the end of this step, the model is bi-dimensional andcannot be animated, deformed, shaped. ASHAPE is the starting point forthe next step.

It will be appreciated that the present invention is not limited to theuse of 93 landmarks for the ASHAPE, but any suitable number can be usedin accordance with the desired resolution and processing power requiredto provide the transformation from the 2D photo to the 3D model oravatar whilst providing a fast response time and hence good userexperience.

The next step is to create the 3D model, step 940. During this step, theclient creates the LP using a “low-mesh” 3D prototype. A “low-mesh” 3Dmodel is a model created using a small number of shapes to minimizehardware requirements, for example, triangles. This prototype is calledthe LP general model (LPGM). Using the LPGM, face-trackedcharacteristics are fitted within the wireframe (step 950), that is, the3D model without applied texture which appears to be made of wiredframes, of the LPGM. This is done by matching some key points of theLPGM with points representing the facial characteristics, which as anon-limiting example can be 93 points, the ASHAPE. All the other pointsare interpolated. This is described in more detail below with referenceto, FIG. 14.

As mentioned above, as a non-limiting example the shape used to createan LP can be made using the 93 landmarks which are grouped together torepresent human or animal facial characteristics. In particular,landmarks O to 4 represent the front, landmarks 5 to 19 the facegeometric outline, landmarks 20 to 25 the right eyebrow, landmarks 26 to31 the left eyebrow, landmarks 32 to 39 the left eye, landmarks 40 to 42the bottom left eyelid, landmarks 43 to 45 the left eyelid, landmarks 46to 53 the right eye, landmarks 54 to 56 the bottom right eyelid,landmarks 57 to 59 the right eyelid, landmarks 60 to 70 the nose, andlandmarks 71 to 92 the mouth.

During this step, ASM is applied to the input image and the client findsall the 93 landmarks to identify the face characteristics of the user.As shown in FIGS. 13A, 13B and 13C, this step is represented in theimage by the points connected by the lines. At (a), the ASM is appliedto the image and then manipulated as shown as (b) until the ASMsubstantially matches the face as shown at (c). The line 1120 (FIG. 11)shows another example of the ASM.

The ASHAPE does not represent a 3D model, but a 2D model made of points,Pi, for each point in the shape, with coordinates (X,Y) on the plane asdefined by:

Pi=(Xi,Yi)

where X is the abscissa, and y the ordinate

As mentioned above, ASHAPE is defined as the set of points or landmarks,as a non-limiting example 93 point, which result from the ASM algorithm.Each point is represented by coordinates P(x, y) having origin in thetop left side of the related image. To complete the process of creatingan LP, one more step is needed where all 93 landmarks are used to fit apreviously generated 3D model to the face.

In order to switch from a 2D model to the desired 3D model, a generic 3Ddeformable model is needed. Such a model is then modified to fit theface by moving some points of the 3D model to match with the points ofthe 2D model.

As the 3D model contains many more points of two-dimensional positionsdefined in ASHAPE, the positions of the remaining points are determinedby linear interpolation. in order to maintain the morphology of theface,

FIG. 3 illustrates high level functionality built into a 3DoM® cloud ormobile device client 300. The user V has access to an activity browsermodule 310, access to a profile manager module 320, the ability tocreate a LP 330, the ability to play with his LP 340, access to a creditmanager module 350, and access to social networking module 360. Inaddition, the user also has the ability to shop, block 370, and playgames, block 380, and can access in app advertising, block 390. Accessto an alerter, block 395, is also provided.

The activity browser module 310 allows the user to navigate throughactivities including but not limited to: received messages,received/sent 3me, and credit status and so on. The activity browsermodule 310 works like an interactive log under the responsibility of theuser and only the user is capable of managing it The profile managermodule 320 provides a facility that allows the user to manage his/her3DoM profile. Using the profile manager module 320, the user is able toenter the social network accounts that he/she wants to use, managehis/her privacy settings, enter personal data and so on.

When creating an LP 330, the user can use an image containing his/herface. This module can connect to the Network Systems to look for a photousing for instance Google® image or to get photos from the socialnetwork user profile. This feature may vary from client to clientdepending on the target platform. If the target platform is an “in appLP engine”, that is, the application is provided with the full set ofAPIs and is therefore self-sufficient; avatars can be created locallywithout connecting to the Network Systems. If the target platform is “LPengine as service”, a connection to the server is needed to create a LPas the application is not capable of creating avatars on its own. TheLP, once created, is sent to the 3DoM® cloud or mobile device server.This is described in more detail below.

In module 340, the user can play with his LP or with the LP of friends.

This is shown in more detail in, FIG. 4 which corresponds to part ofFIG. 2. Elements that have been described with reference to FIG. 2 aresimilarly numbered but bear the prefix ‘4’ instead of ‘2’. In sub-module436, the user can customize his/her LP by adding apparels like hats,wigs, glasses or other kind of gadgets. Clothing, 3me® (or MoodEe®) andgadgets can be downloaded from the 3DoM® online shop. These may becharged to the user or may be free depending on the profile of theindividual user. In sub-module 438, the user can change the make-up ofthe avatar.

In sub-module 434, the user can use expressions to create differentmoods. Here, the user can change the facial expression of his/her LPusing a pre-compiled set of expressions. More precisely, a facialexpression is defined as being a feature that shows an emotion and whichlasts only a few seconds. For example, a smile appears almost instantly,but, in the next moment, the face is still smiling but not with the sameintensity, and, a few seconds later, the face stops smiling. Changingthe expression of the LP does not change its status on the cloud ormobile device server. This is because expressions are not uploaded to ormodified through the cloud or mobile device in order to save resourcesand cost, for example, data bandwidth used and processing power.However, if sufficient data bandwidth and/or processing power can bemade available at a reasonable cost to the user, expressions may also beuploaded to the 3DoM® Platform for sharing through the cloud or mobiledevice.

Expressions can be triggered by an external event, for example, a missedcall, low battery, and the like.

In module 432, the mood of the LP can be changed. Moods are persistentand complex. They may be defined as a set of expressions, dress up, makeup, and the like. This means that when the user changes his/her mood,his/her LP behaves accordingly. The mood is uploaded by the user'sclient device 124, FIG. 1(b), that connects and sends a messagerepresenting the mood update to the 3DoM® Platform hosted on third partyprovider 120, FIG. 1(b), (step 210), which allocates a resource 122,including but not limited to a CPU and memory, to process it, to log itwith the Archive Component 144, FIG. 1(e). The 3DoM Platform notifiesthe user's client devices through an Interface Component 126, FIG. 1(b).If the users to be notified are not available (for instance thesmartphone is switched off or not connected to a Network System), theywill be notified the next time they connect to the 3DoM Platform. In oneembodiment, some moods are simple, for example, displaying happiness,tiredness, boredom, angriness etc. or more complex, for example, “I'm apirate” and “I feel like a zombie”. All the client devices are preloadedwith a default set of moods. The user also can create his/her own moodsand upload them to the cloud or mobile device. This is described in moredetail below.

Returning to FIG. 3, the credit manager module 350 allow a user topurchase apps etc. using a virtual currency, in this case, 3DoM® dollars(3DoM$). However, this may be amended in cases, such as, iPhone®applications, where the user can only use payment gateways through theApple® on-Istore or other store associated with the mobile device. {BothiPhone® and Apple® are trademarks of the Apple Corporation.) Creditmanager activities are shown in FIG. 5.

In FIG. 5, a credit manager system 500 is shown. As described above, theuser connects to 3DoM using his/her social network account, step 510.From there, he/she can get credits (step 520), manage credits (step530), trade credits {step 540) and spend credits (step 550). Naturally,other options are also possible.

FIG. 6 illustrates the way to get credits in more detail. As describedabove elements that have previously described with reference to FIG. 5have the same reference numerals. The user may obtain credits in manyways, but only two ways are shown as examples. Credits can be boughtusing real money (step 610), or can be awarded as rewards for activitiesconducted in the 3DoM® world (step 620). Such activities include:inviting friends to join 3DoM®, publishing invitation messages onto thesocial network, conducting advertising activities, etc. However, allactivities conducted to get rewards follow predetermined rules and needto be verified. This means that all the rewarded activities areperformed by the 3DoM® cloud or mobile device client and the user isguided to choose between the activities proposed by the system. A listof rewarded activities is provided by the system (step 630), the userselects one of the activities (step 640), and performs the activity(step 650) to get the rewards in step 620.

Returning to FIG. 5, a facility, manage credits (step 530), is providedto enable the user to check his/her credits, and to access his/her logof activities (spent credits, rewards, etc. etc.). Through thisinterface, the user has the ability to report potential fraud and/orunauthorized access to his/her account and/or use of his/her credits.

Another facility, trade credits (step 540), provides a way for a user toearn credits by “selling” self-created gadgets or moods so that they areloaded onto the store hosted on the 3DoM® Platform. Naturally, beforesuch items are loaded onto the 3DoM® Platform, they would need to betested to determine their usefulness. This potentially has severaladvantages. For example, the most creative users are retained, thenumber of gadgets and moods available on the system are increased, andthird party companies are rewarded for creating products for the 3DoM®Platform.

Similarly, spend credits (step 550) allow the user to spend his/hercredits by buying gadgets, 3me, new moods and so on.

As described above, the 3DoM® Platform of the present invention does notsubstitute already existing social networks. The present inventionintegrates seamlessly with such networks through the user's socialgraph, which is accessible via an application programming interface(API) provided by the referenced platform. This has many advantagesincluding: instant access to connected users, access to user data (inaccordance with privacy rules set by the user), letting social networksmanage their users' graphs, and the opportunity to be more effectiveduring a marketing campaign. As a non-limiting example, using the socialgraph provided by the social networks means cost saving when storinguser data onto the 3DoM® cloud or mobile device servers.

FIG. 7 illustrates a social networking activity diagram 700 whichillustrates the activities for which the social networking module isresponsible. As before, the user ‘U’ connects to the 3DoM® Platformusing his/her social network account 720 and, after user authentication,step 715, has access to various activities as shown. Userauthentication, step 715, is responsible for performing userauthentication for the 3DoM Platform through his/her social networkaccounts. This allows the user to connect to one or more social networksusing account data he/she provides. All the user account data is storedon the 3DoM® cloud or mobile device server database to allow the 3DoM®Platform to share all the authentication information between the userclients.

One critical module is cross social network integration as indicated by730. Cross integration between social networks does not mean that the3DoM® Platform fuses different social graphs from different socialnetworks, but provides high level interfaces to implement cross socialnetworks functionality and to provide a set of functionalities that canbe used by all client modules. A social graph for an individual on asocial network or networking site refers to that individual and his/herassociated connections. The operation of this module is described inmore detail below with reference to FIG. 8.

Sharing of LP facility, block 740, is the core of the 3DoM® Platformsharing of emotions as all the other functionality elements rely uponthe possibility of sharing moods and emotions of the LP with connectedfriends through the 3DoM® cloud or mobile device server. This isdescribed in more detail below.

In a “send 3me®” module 750, a virtual object is sent to catch theattention of the LP of a recipient. To do this, the client device 124,FIG. 1(b), sends a message containing the 3me to the 3DoM platformhosted by third party service provider 120 who allocates a resource 122,including but not limited to a CPU and memory, including but not limitedto a CPU and memory, to process it The sending of the 3me is then loggedin a log file managed by the archive component 144, FIG. 1(e), and sendsthe 3me to the recipient through an interface component 126, FIG. 1(b).Depending on the embodiment, some 3me require explicit acceptance by therecipient, in which case the acceptance of the 3me is notified by therecipient client device to the 3DoM Platform and the acceptance islogged as before.

3me® (or MoodEe®) are complex objects which can be purchased from thestore, and produce effects that are applied to the LPs. As non-limitingexamples, a 3me® (or MoodEe®) can be a kiss, a slap, a wasp bite, aglass of a drink, a flower bouquet, etc. In addition, a 3me® (orMoodEe®) can contain a text which is composed at the moment of sendingthe 3me® (or MoodEe®) or which is already available. When the userreceives a 3me® (or MoodEe®), the client notifies him/her and asks theuser if the 3me® (or MoodEe®) is accepted or not. If the 3me® (orMoodEe®) is accepted, a message is sent to the sender and the 3me isstored locally. 3me (or MoodEe®) (always in singular, just like SMS) canbe used to provide emotions to friends by animating their LP anddelivering text messages to them.

Upon receipt of the 3me® (or MoodEe®), the LP of the recipient candisplay one or more of, as non-limiting examples: a facial expressionfor example, a smile, surprise, sadness, and the like; skin textures,for example, stamping red lips on the face for a kiss; greening of theskin for a bite by a wasp; bruising the skin for a slap etc.; agraphical display of the object forming the 3me® (or MoodEe®); and atext message which is displayed on the screen. If the receiver acceptsthe 3me® (or MoodEe®), the user is offered the possibility of sendingback a message with an included expression to notify the sender ofhis/her feelings on receipt of the accepted 3me (or MoodEe). If the 3me®(or MoodEe®) is refused, a message is sent to the sender and the 3me®(or MoodEe®) is deleted from the receiver client

Module 760 allows the user to get LPs of his/her friends. As describedabove, the 3DoM Platform introduces new ways to communicate with friendsthrough the cloud or mobile device using social networks. A 3DoM® userstays connected with his/her friends through their LP. This featureallows the user to be able to check the moods of his/her friends, to benotified about the change in the mood of a friend, to communicate withconnected friends, and to play with the LP of friends as shown by module765. This kind of connection can be done by navigating the “socialgraph” of the social network used to connect to the service.

In module 770, messages can be sent to friends. This provides thepossibility to share moods and expressions with them and to introduce anew way to communicate with connected friends. Using the 3DoM® Platform,the user can send a message to a friend as part of a 3me® (or MoodEe®)which includes his/her feelings when writing the content. In this way,an LP can now become an advanced version of an emoticon making the userable to “virtually” understand the mood of the message sender. Themessage being sent as part of a 3me® (or MoodEe®) may be displayed in anoriginal way, such as, a bubble spoken by the LP like in comic books, ora text-to-speech conversion using a third party technology, etc. 3me®(or MoodEe®) with messages can be considered to be an evolution of thecombination of SMS and emoticons.

3me® (or MoodEe®) objects may fall within one of three categories: thosethat are free; those that are charged; and those that are sponsored.Sponsored 3me® (or MoodEe®) will be freely available for the userssending them, as they will be sponsored by companies doing advertisingcampaigns, for example, a glass of vodka, sponsored by Smirnoff® or achocolate bar sponsored by Cote d'Or®. Smirnoff® is a trademark of theDiageo pie, and Cote d'Or® is a trademark of Kraft Foods. The recipientof the 3me® (or MoodEe®) will see a fun animation of expressions onhis/her LP relating to the object and will be able to receive a realversion of that object in a participating store or bar.

Through the 3DoM® client, the user can change his profile photo on hissocial network, module 780. As an example, a user can change theexpression of his LP or the dressing of the LP, and then can substitutethe social network profile photo with a new one representing a snapshotof the LP. At each moment, the user can share his mood by changing thesocial network profile photo according to his/her feelings at thatmoment.

To perform this, the user's client device 124, FIG. 1(b), connects andsends a message representing the request to update his profile pictureon a given social network to the 3DoM® Platform hosted on third partyprovider 120, FIG. 1(b), (step 210), which allocates a resource 122,including but not limited to a CPU and memory, to process it. The 3DoMPlatform connects to the Social Network on behalf of the user throughthat social network API and an Interface Component 126, FIG. 1(b), toperform the request. The 3DoM Platform notifies the user back of thesuccess or otherwise of the request through and an Interface Component126, FIG. 1(b), and logs it with the Archive Component 144 (FIG. 1(e).

Also through the 3DoM® client, the user can publish his/her LP directlyon his/her social network page/wall with a comment, module 790.Depending on the construction of the social network page/wall, the 3D LPmay be converted to a 2D representation thereof. This publicationfollows similar steps involving the user client device, the 3DoMPlatform and the Interface Component.

FIG. 8 illustrates an interface module 800 for cross social networkintegration. The interface module 800 comprises a cross network platform810 with a plurality of programming interfaces 820 that connect torespective ones of functions 830, 840, 850, 860, 870 on the platform810. Such functions include but are not limited to, authentication,sending messages, publishing photos, changing profile photo and variousother functions. The network platform 810 also interfaces with theassociated APIs 880, 885, 890 of the social networking sites, such as,Facebook®, Twitter®, Google+® and LinkedIn®. LinkedIn® is abusiness-related social networking site.

Advertising is one of the sources of revenue for any social networkingsite and the 3DoM Platform is no exception. A dedicated module managesadvertising inside the client. A dedicated library provides the abilityto display advertising that takes advantage of all the user informationso that the most relevant advertising can be selected for display. Themood of a user may significantly influence his/her receptiveness to anadvertisement of a given product or service. Knowing that a user ishappy, hungry, thirsty, hot or cold could be an indication that he/shemay well be disposed to receiving an appropriate advertising message.The 3DoM® platform is aware of the mood of its users and of theirchanges in near real-time as the client devices 124, FIG. 1(b), sendmessages with information about LP mood and other updates (such asaccessories used by the LP, etc.) to the 3DoM Platform hosted by a ThirdParty Service Provider 120, FIG. 1(b). Such information is processed byallocating resource 122, including but not limited to a CPU and memory.This information is stored in a local Data Store 146, FIG. 1(e).

The 3DoM® platform makes information relating to the mood of its usersavailable to third parties such as advertisers and information providersthrough a dedicated library (API) which will offer a unique targetingsystem: ads and more generally information such as news will be selectedbased on people's mood. Users of 3DoM will also be able, through thesetting of personal preferences, to opt in & out of different types ofadvertising and news, or of all information altogether.

Third Party advertisers and other content providers (from now on ContentProvider) may, provided that they have a commercial agreement withMoodMe, connect with the 3DoM Platform through an API and an InterfaceComponent 126, FIG. 1(b), to request information about Users who had aspecific Mood at a very recent moment or other characteristics asspecified by the Content Provider. These requests are processed byallocating resources 122, including but not limited to a CPU and memory,they are logged in the Archive Component 144 (FIG. 1(e)) and theinformation requested is sent back to the Content Provider. ContentProviders may submit content (advertising or other) to MoodMe users whomatch the Mood state or other characteristics through the API and theInterface Component 126, FIG. 1(b). The 3DoM Platform will retarget thecontent to the users' client device 124, FIG. 1(b), through theInterface Component 126, FIG. 1(b). It will also log this in the ArchiveComponent 144 (FIG. 1(e)) and will compute billing elements to chargethe Content Provider for the service.

In one embodiment, the creation of an LP requires the ability totransform a two-dimensional (2D) human or animal portrait into a 3Ddeformable model.

FIG. 9 illustrates a flow chart 900 of the steps required to create a 3Dface model from a 2D image of a human or animal face thereby creating aLP. The first step is image acquisition and digital elaboration, step910. Here, the client acquires images from different sources. If needed,the algorithm applies image filters to enhance the image quality (notshown). The next step is to search for a human or animal face inside theimage, step 920, and the client restricts the operation of the next stepof the algorithm by finding an area containing a human or animal face.Once a human or animal face has been identified, a search for facialcharacteristics is carried out, step 930. The search is restricted tothe previously detected area which contains a human or animal face. Inparticular, the algorithm looks for the following facialcharacteristics: front, eyes, eyebrows, eyelids, face geometric outline,nose and mouth. Each characteristic is then marked using a set ofpoints, or landmarks, each landmark being uniquely identified by anumber as will be described in more detail below. Marking can be used tofind the points and can be achieved using the ASM algorithm.

In FIG. 10, the image acquisition and digital elaboration step 910 isshown in more detail. For the first step (step 1000), the user acquiresa photo. As non-limiting examples, this can be done using a webcam (step1010), searching onto the Network Systems (step 1020) or simplynavigating into the mobile handset image gallery (step 1030).Alternatively, the photo can be uploaded from a local file system (step1040).

When using the Network Systems as the photo source, the photo can beloaded from a social media site, including but not limited to Facebook®(step 1022), from a Google® search (step 1024) or by inserting an imageURL (step 1026). To support this phase, the application may select onlyphotos that contain a human or animal face seen substantially from thefront, or “face-on”. However, it may also be possible to select a photofrom a slight angle to either side from the “face-on” position.

During loading of the photo, the application automatically applies someimage filters {step 1050) to enhance the quality of the photo. In oneembodiment such filters are used to regulate image contrast or expandcolor gamma that is the range of the visible colors. The application offilters improves the result and enhances image quality, as well asprecision, when creating an LP.

Histogram equalization is a method that is used in conventional imageprocessing for adjusting image contrast of the original image. Gammacorrection controls the overall brightness of an image. Images that arenot properly corrected can look either bleached out or too dark. Tryingto reproduce colors accurately also requires some knowledge of gamma.Varying the amount of gamma correction changes not only the brightnessbut also the ratios of red to green to blue. In one embodiment improvingthe quality of an image refers to working on the general parameters ofbrightness and contrast to enhance the image quality, and nothing can bedone to create “information” if the image is low quality.

Before proceeding with the recognition of facial characteristics, step,1060, a determination is made of an approximated area containing humanor animal face and estimate the starting pose alignment. This can beachieved with the ASM algorithm disclosed herein. Good pose estimationprovides a good starting point and allows the processor and memory thatexecutes a processor and memory that executes software, including butnot limited to algorithms, including but not limited to algorithms, tominimize resources used in the creation of the 3D model, for example,battery usage for mobile devices. This image area is generally called aregion of interest (ROI). In FIG. 11, an image 1100 is shown in whichthe estimated area is represented by a rectangle 1110 which surroundsthe points determined as facial characteristics 1120.

One embodiment offacial recognition is disclosed in U.S. Pat. No.8,254,691, fully incorporated herein by reference.

The ROI may be automatically executed by the processor and memory thatexecutes software, including but not limited to algorithms, using Haarlike classifiers, a tool to determine if a determinate pattern may befound inside an image-squared region, or manually by the user. If donemanually, as shown in FIG. 12, the user then moves, scales, and rotatesthe image using the icons on the toolbar 1130 until the face is alignedto a white shape 1140 used as guideline. A rectangle (not shown)containing the white shape 1140 becomes the ROI for the image.

Returning to FIG. 9, facial characteristics are determined in step 930,as described above. This step represents the core of the whole processof transforming a 2D image into a 3D model. The higher the accuracyachieved at this stage, the better the end result.

In one embodiment, an ASM (Active Shape Model) algorithm is used, whichis based on statistical models of shapes of objects, to identify thelandmarks on the image. The process of fitting of a model to a faceinside an image is shown in FIGS. 13A, 13B and 13C.

FIG. 11 illustrates one embodiment of the execution of the ASMalgorithm. The ASM converts color images into gray scale images for thepurpose of finding the landmarks.

In one embodiment, the ASM s is used with a level of detail so that theASHAPE is composed of 93 landmarks, which provides a compromise betweengood quality of the 3D model or avatar produced and fast response timeof the application providing a better user experience

In one embodiment, in order to create a 3D deformable model of the humanor animal face, a wireframe generic 3D model of human or animal face iscreated using a 3D graphics technique specific to the present invention.

As a non-limiting example, the 3D model is created in one embodimentsuch that it has:

-   -   Portability and can be used on all platforms with 3D graphics        capability by using a 3D model format based upon the XML open        standard;    -   Good definition and high quality of the rendering of facial        characteristics so that the face is realistic and respectful of        the person's morphology;    -   Fast animation that provides, as a non-limiting example, 20 or        more frames which can be generated per second, in order to        generate cinematographic quality of the movements of the face;    -   Low battery consumption by optimizing the execution of the code        on the client device;    -   A data size of the 3D model below 500 KB so that it can easily        be transmitted via email or via a Network Systems connection        without adversely affecting the minimum acceptable quality        level.

FIG. 14 illustrates a low mesh face mask specifically developed formodel-based coding of human or animal faces adaptable to the 2D facemodel made of 93 landmarks which describe human or animal facialcharacteristics. For simplicity not all the points and vertex are shownin the Figure.

As shown in FIG. 14, the model comprises a number of points or vertices.These points or vertices are either anchor points (AP) or simple points(SP). Anchor points are a set of special points in a 1-to-1 relationwith points within the 2D model. All anchor points define a set calledAPS. Simple points are necessary to animate the model and enhance thequality of the 3D model. All simple points define a set called SPS. As aconsequence, an LPGM is the set of all the AP and SP and can berepresented by:

LPGM=APS_,SPS

-   -   Where APS=ASHAPE

The points in the LPGM can be represented as coordinates in space as P,=(X1i, Y1i, 21i), and these coordinates may be represented by theirhomogeneous form where:

P1_(i)=(Xi,Yi,21,w)=(X/w.Y/w,2/w,1)=(X1_(i) ,Y1i;21_(i),1)

The LPGM is made of three sections which describe respectively, the listof vertices, the list of faces and a list of forming units (FU).

The list of vertices is used to define the corners of surfaces,typically triangles, in 3D models where each such point is given as avector. In one embodiment, the vertices are represented using a vectorR3, which can be expressed as (Vx, Vy, Vz). In the following example,the X, Y and Z coordinates are shown of a single vertex represented inXML format:

<LPVERTEXUST>   <LPVERTEX ID= “FOREHEAD_1 ”>     <X>0.174</X>    <Y>0.98</Y>     <Z>−0.024</Z>   </LPVERTEX> </LPVERTEXLIST>

As a non-limiting example, in one embodiment triangles are used todefine polygonal meshes, connected vertexes and primitives. A polygonalmesh, connected vertexes and primitive is a collection of vertices,edges and faces that defines the shape of an object in 3D computergraphics and solid modeling. In one embodiment, the faces are triangles,quadrilaterals, simple convex polygons but they can also be made frommore general concave polygons, or polygons with holes. It will beappreciated that other mesh structures with vertex's can also beutilized.

the vertices represent a generic face model centered in the origin ofaxes in the model shown in FIG. 14. As a non-limiting example there canbe 150 vertices where 93 vertices correspond to the landmarks determinedpreviously. It will be appreciated that different number of vertices canbe utilized.

In another embodiment, the mouth can be represented by 26 verticesinstead of 22.It will be appreciated that different number of vertices can beutilized: adding vertices will allow to create higher definitions modelswhich will be more realistic and will also require more computingresources to be rendered and animated.

It will be further appreciated that other geometries are possible, aswell as combinations of combinations, and that some of the vertices canbe formed from non-linear lines.

After ASHAPE has been found using the ASM algorithm, the next step isfitting the LPGM to conform to the ASHAPE. At the end of this operation,some points in LPGM, namely, points in APS, are aligned to the points ofASHAPE. All the other points in LPGM, namely, points in SPS, aremanually evaluated or interpolated to find their correct position, forexample, by matching some key points of the LPGM with pointsrepresenting the facial characteristics (93 points as described above),the ASHAPE.

Because the ASHAPE is a 2-dimensional model and the LPGM is a 3D model,points in LPGM are translated, that is, moved from one position toanother one, only along the X- and Y-axes. Because the original imagedoes not contain information about depth of the face, the Z-axis isevaluated empirically on the basis of information from the ASHAPE.

The master model is now aligned to the ASHAPE. In the next step, a skinneeds be created that will be applied to the model. Here, the model skinis created using the information from the original image. To do this,simple texturing is used—a basic method for mapping a texture onto anobject. In one embodiment, this can be done using open graphic libraries(OpenGL) API which is a reference graphic subsystem for all the mobiledevices and desktop computers. The method of creating textures isdescribed below with reference to FIG. 15.

In one embodiment the list of meshes are used for texturing and shadingin the 3D model. In one embodiment these meshes are triangles, and othergeometric shape, including linear and non-linear components, thatachieves a similar result as the triangles. Each mesh is represented bythree or more vertices and is listed by its ID as it appears in the listof vertices. Each mesh or triangle is used to create textures from theshades to render the 3D model. Vertices are listed using their ID as itappears in the vertex list. For a single face:

In the following example, a mesh composed is listed of three vertices,all three points of the MOUTH, represented in XML format:

<LPFACEUST>   <LPFACE ID = “T_MOUTH_UP_INFERIOR_1”>     <V1 >MOUTH_1 <N1 >     <V2>MOUTH_2< N2>     <V3>MOUTH 3<N3>   </LPFACE> </LPFACELIST>

The list of forming units represents the central element for theanimation of the model and the representation of expressions. The nameof each item represents the action when applied to the model; an index(INDX) represents the index of the vertex or vertices affected by thisFU; and convolution values (COV) are values that need to be applied tothe coordinates (x, y, and z) of the vertex during the affine transform.Whilst the LP has z coordinates, these are not used when skinning ortexturing as these only need features defined by a 2D surface.

An affine transform or map is a transformation which preserves straightlines and ratios of distances. In particular, all points initially lyingon a line still lie on the line after transformation and the midpoint ofa line segment remains the midpoint after transformation. It is to benoted that an affine transform does not necessarily preserve angles orlengths.

Referring now to FIG. 15, a flow chart 1500 of steps needed for creatinga model texture is shown. In order to create the skin of the LP, thebase image points are used, step 1510. For each point in the LP, thecorresponding point projected on the x, y plane is determined as follows(step 1520):

V P=(x,y,z)e LP=>Pxy=(x,y,0)

For each face in the list of faces, an image area corresponding to therelated triangle strip is created (step 1530), and the set of thetriangular image strips represent the skin of the LP. These triangularstrips are then cut in step 1540 before reconstruction in step 1550. Thethird step is the application of the textures to the model of the face.Each triangle generated is then applied to the corresponding faceportion and is controlled by the graphic engine of the device on whichthe LP is being rendered. In the last step of LP creation, it isnecessary to reconstruct the non-visible parts of the face, namely, theinterior of the mouth, the tongue, the throat, the teeth etc. asdescribed above.

The final LP can be shown as facing a source of light. However, thelight source can be moved sideways, up or down, so that shadows appearon the face of the LP providing a more realistic 3D appearance. Movementof the head of the LP causes the shadows to be adapted with the movementin accordance with the direction of the source of light.

The final LP can be defined as a triple combination of the base image(the image used to generate the ASHAPE), the ASHAPE (the 2D model of theface) and metadata (a list of metadata describing the LP).

Now the model is ready to be animated. There are two different types ofanimation, namely, automatic animation and user-induced animation.Automatic animation gives “life” to the model and is independent of userinteraction, whilst user-induced interaction provides animation due tothe interaction of a user with the model to give feelings and facialexpressions. These two types of animation can be coupled to obtain athird kind of animation which gives a better representation of the moodof the LP.

As stated previously, the model is a 3D model comprising vertices andfaces with a texture mapped onto them. After the model has been fittedto the human or animal face, it can be represented by a 3N-dimensionalvector where N is the number of vertices as represented following:

¢>=(x1,y1,z1,x2,y2,z2, . . . xn,yn,zn)

where (xi, yi, zi) represents the coordinates of the i-the vertex.

Animating the 3D model consists of re-shaping the model by translating,rotating, scaling vertices. For example, a user input of touching thescreen makes the head of the LP look towards the point of contact on thetouch screen. In the absence of a manual input, the head can be swungaround its point of rotation using a simulation of natural movements.This simulation is based on rotation about the X-, the Y- and theZ-axes.

Rotation about the X-axis relates to movement around the axis passingthrough a pivot point (as defined below) and which is parallel to theX-axis. The angle of rotation may be predetermined and when reached, anew value is set, or, alternatively, the angle of rotation is notpredetermined. The new angle of rotation, Axi, can be randomly chosen ina range of values defined by intersections of sets. Ax1 can be definedas the set of angles between ±5°; Ax2 can be defined as the set ofangles having a distance from the current angle of rotation in the rangebetween 7.5° and 30° with 0° set as a default for the first time; andAx3 can be defined as the set of angles having a sign that is oppositeto that of the current rotation angle with 0° set as a default for thefirst time. A new angle of rotation, Axr, can be defined as thefunction, R(Ax}:

Axr=R(Ax1

Ax2

Ax3)

Similarly, rotation duration, Txr, can be associated with the new angleof rotation, Axr, along the X-axis. The rotation duration can beexpressed in ms: where Axr1 is the next chosen rotation and AxrO is thecurrent completed rotation, a minimum rotation duration, Min{Txr), and amaximum rotation duration, Max(Txr), can be expressed respectively asfollows:

Min(Txr)=100 ms×distance between AxrO and Axr1 (in°)

Max(Txr)=200 ms×distance between AxrO and Axr1 (in°)

In addition, Txr can be defined as the value, randomly chosen from theset of values in the range defined by the previously defined functions:

R(Min(Txr)Txr Max(Txr))

Together with the angle and the time of rotation, a function is used toease the movement. Five different easing functions have been defined,namely: LINEAR, QUAD, CUBIC, QUART, QUINT. The algorithm selectsrandomly from the set of all easing functions as follows:

EFxr=Random({UNEAR,QUAD,CUBIC,QUART,QUINT})

Having defined Axr, Txr, and the easing function, until the new limit isreached, that is, before choosing another angle and time of rotation,part of the movement is done applying EFxr, and rotation, ROTx, aboutthe X-axis can be defined as:

ROTx=(Axr,Txr,EFxr)

Similarly for rotation around the Y-axis, Ay1 can be defined as the setof angles between ±12°; Ay2 can be defined as the set of angles having adistance from the current angle of rotation in the range between 4.8°and 24° with 0° set as a default for the first time; and Ay3 can bedefined as the set of angles having a sign that is opposite to that ofthe current rotation angle with 0° set as a default for the first time.A new angle of rotation, Ayr, can be defined as the function, R(Ay):

Ayr=R(Ay1

Ay2

Ay3)

As before, rotation duration, Txy, can be associated with the new angleof rotation, Ayr, along the Y-axis. The rotation duration can beexpressed in ms: Where Ayr1 is the next chosen rotation and AyrO is thecurrent completed rotation, a minimum rotation duration, Min(Tyr), and amaximum rotation duration, ax(Tyr), can be expressed respectively asfollows:

Min(Tyr)=100 ms×distance between AyrO and Ayr1 (in°)

Max(Tyr)=300 ms×distance between AyrO and Ayr1 (in″)

In addition, Tyr can be defined as the value, randomly chosen from theset of values in the range defined by the previously defined functions:

R(Min(Tyr)Tyr Max(Tyr))

Together with the angle and the time of rotation, a function is used toease the movement. Five different easing functions have been defined,namely: LINEAR, QUAD, CUBIC, QUART, QUINT. The algorithm selectsrandomly from the set of all easing functions as follows:

EFyr=Random({LINEAR,QUAD,CUBIC,QUART,QUINT})

Having defined Ayr, Tyr, and the easing function, until the new limit isreached, that is, before choosing another angle and time of rotation,part of the movement is done applying EFyr, and rotation, ROTy, aboutthe Y-axis can be defined as:

ROTy=(Ayr,Tyr,EFyr)

Similarly for rotation around the Z-axis, Az1 can be defined as the setof angles between ±2.2″; Az2 can be defined as the set of angles havinga distance from the current angle of rotation in the range between 4.8°and 24° with 0° set as a default for the first time; and Az3 can bedefined as the set of angles having a sign that is opposite to that ofthe current rotation angle with 0° set as a default for the first time.A new angle of rotation, Azr, can be defined as the function, R(Az):

Azr=R(Az1

Az2

Az3)

As before, rotation duration, Tzr, can be associated with the new angleof rotation, Azr, along the Z-axis. The rotation duration can beexpressed in ms: Where Azr1 is the next chosen rotation and AzrO is thecurrent completed rotation, a minimum rotation duration, in(Tzr), and amaximum rotation duration, Max(Tzr), can be expressed respectively asfollows:

Min(Tzr)=100 ms×distance between AzrO and Azr1 (in°)

Max(Tzr)=500 ms×distance between AzrO and Azr1 (in°)

In addition, Tzr can be defined as the value, randomly chosen from theset of values in the range defined by the previously defined functions:

R(Min(Tzr)Tzr Max(Tzr))

Together with the angle and the time of rotation, a function is used toease the movement. Five different easing functions have been defined,namely: LINEAR, QUAD, CUBIC, QUART, QUINT. The algorithm selectsrandomly from the set of all easing functions as follows:

EFzr=Random({LINEAR,QUAD,CUBIC,QUART,QUINT})

Having defined Azr, Tzr, and the easing function, until the new limit isreached, that is, before choosing another angle and time of rotation,part of the movement is done applying EFzr, and rotation, ROTz, aboutthe Z-axis can be defined as:

ROTz=(Azr,Tzr,Fzr)

Pivot Point

The origin of the three axes of rotation, the so-called pivot point, isautomatically determined after the fitting of LPGM on ASHAPE. This pointis obtained using the X- and Y-coordinates of the sixty-sixth landmarkof the model LPGM. The Z value of the required point is equal to the11/10 of the depth of the model.

The easing function is used to vary the speed of the rotation during theexecution of the movement. All easing functions (LINEAR, QUAD, CUBIC,QUART, QUINT) are based on 3 arguments, namely: D, the duration of theanimation in seconds; C, the final absolute size of the movement; and T,the time in seconds spent to go from O to D on which the value C isevaluated. Therefore, the LINEAR easing function can be defined asUNEAR(D, C, T)=(C*T)/D; the QUAD easing function as:

QUAD(D,C,T)=C/2*T ²(if (T=D/2)<1)

QUAD(D,C,T)=−C/2*(T1*(T1−2)−1)(where T1,=T−1)

Similarly, the CUBIC, QUART AND QUINT easing functions can be definedas:

CUBIC(D,C,T)=C/2*T ³(if(T=D/2)<1)

CUBIC(D,C,T)=−C/2*(T ³+2)(where Ti=T−2)

QUART(D,C,T)=C/2*T ⁴(if(T=D/2)<1}

QUART(D,C,T)=−C/2*(T ⁴−2)(where Ti=T−2)

QUINT(D,C,T)=C/2*T ⁵(if(T=D/2)<1)

QUINT(D,C,T)=−C/2*(T ⁵+2)(where Ti=T−2)

Not all the vertices in an LP are moved according to the defined rules.To make the LP animation more realistic, the LPGM has been structured tocover not only the face geometric outline but also an area containingthe image background. From the animation point of view, the LPGM isdivided in many areas which are concentric rings;

-   -   Contour 3, the outer edge, far from the head geometric outline,        including large portions of the background;    -   Contour 2, the area between Contour 3 and the face geometric        outline, including hair and small portions of the background;    -   Upper Contour, an upper portion of Contour 2, including all        points on the upper side of the eyebrow; and    -   Lower Contour, a lower portion of Contour 2, including all        points on the lower side of the chin.

To evaluate X-, Y- and Z-axis rotation to be applied to each vertex ofthe LP, the following rules are applied:

(1) if a vertex belongs to Contour 3, no rotation is applied therebyanchoring all points in Contour 3 to the image background;

(2) if a vertex belongs to Lower Contour, X- and Y-rotations are reducedby 50% of their value with the Z-axis rotation being reduced to 10% ofthe original value;

(3) if a vertex belongs to Upper Contour, X-, Y- and Z-axis rotationsare reduced to 60% of their original value;

(4) if a vertex belongs to Contour 2, X-, Y- and Z-axis rotations arereduced to 30% of their original value; and

(5) all other vertices are moved according to the defined rules.

Note: These values are given as example only and can be modified indifferent implementations. The principle is to have different level ofrotation for areas around the face in order to generate a more realisticanimation of the head rotation.

This kind of movement, especially between Contour 2 and Contour 3,introduces very noticeable deformations. Because Contour 3 is fixed, allthe textures are stretched with a disturbing effect. To reduce thiseffect, a gradient is applied, which gradually makes textures completelyopaque in the area next to the center of the LP, and almost totallytransparent in the area next to the top of Contour 3.

Expressions

Expressions are little animations induced by the user or automaticallygenerated by the application that represent human or animal facialexpressions. Each expression is made of small actions that concur tocreate a complex animation. The simplest action in facial expression ismoving a simple vertex by applying the affine transform to translate thevertex from one position to another.

Applying the following LP Forming Unit (LPFU), the movement of theraising of the 10th vertex of the LP by a factor defined by the vector(COV1, COV2, COV3). This example shows the movement of the 10^(th)vertex of the LP, belonging to the MOUTH, with the values of the vector,represented in XML format:

< LP ACTIVE VERTEX >    <INDX>MOUTH_10</INDX>    <COV1 >0.0</COV1 >   <COV2>0.086957</COV2>    <COV3>0.02 739</COV3> </LP ACTIVE VERTEX >

Complex animations groups the movement of different vertices logicallygrouped to form meaningful movement. Those groups of movements arecalled LP Forming Unit (LPFU). LPFU defines a movement of one or morevertex of the LP to form an expression. As an example, applying thefollowing LPFU, the upper lips of the model can be raised. Thesemovements have been developed in the course of complex and iterativetrial and errors exercises and are all documented in tables.

However, applying one or more LPFUs is not sufficient to create anexpression. This is because each LPFU statically changes the position ofone or more points from point A to point B without a transition betweenA and 8. In order to create a particular expression, a storyboard iscreated for that expression. Such a storyboard 1600 is shown in FIG. 16.

To animate the model, each LPFU must be applied following a storyboardthat takes into account the LPFUs, their order, the duration of eachLPFU, the easing functions, and the reverse mode to take the model backto the main expression. In FIG. 16, the LPFU is loaded from the LPmodel, step 1610. In order to complete the animation of an expression,animation handlers are used that group one or more LPFUs, that is, allthe information necessary to animate the LPFU as well as their sequence,step 1620, and the expression is executed over a time period (of theorder of a second) by playing the storyboard, step 1630.

The LPFUs are grouped into “Animation Bundles” that represent theanimation of a single LPFU {a single event into a storyboard).

Each “Animation Bundle” is preferably provided to the user on the screenof his/her mobile device so that he/she can make an appropriateselection to animate his/her 3D model or any other 3D model to whichhe/she has access.

In the following example, the method for the “Lip Stretcher” creates ananimation bundle, adds the “Lip Stretcher” LPFU to it and then insertsthe Animation Bundles inside the expression storyboard.

   PUBLIC VOID LIPSTRETCHER( ) {      ANIMATIONBUNDLE BUNDLE= NEWANIMATIONBUNDLE{);    BUNDLE.ANIMATIONS.ADD(PREPAREANIMAT10NFORACTIONUNIT(“LIP STRETCHER”, 0, −110F,0, 1000, FALSE, PICKRANDOMEASING( )));    BUNDLE.ANIMATIONS.ADD(PREPAREANIMATI0NFORACTIONUNIT(“LIP STRETCHER”, 0, −1 1OF,    2000, 1000, TRUE, PICKRANDOMEASINGO));    SYNCHRONIZED(ANIMATIONBUNDLES) {       ANIMATIONBUNDLES.ADD(BUNDLE);       }    }

Sentiments can be extracted from textual data using natural languageprocessing, text analytics and semantic technologies identifyingpositive and negative opinions, emotions, and evaluations, and takingthe context into consideration to disambiguate interpretations.

Extracting indicators about opinions and emotional reactions of thewriter or speaker mean that it can then be expressed with the display ofemotions on the LP of the user. In one embodiment the present inventionuses techniques and approaches that enable directly opinion-orientedinformation-seeking systems allowing the programmer to get into useropinion using dedicated sets of AP!.

In one embodiment, there is a functionally of sharing LPs, moods andemotions through the structure as illustrated in FIGS. 1(a)-1(e). Aclient device, such as a mobile device (or a desktop or other clientapplication) disclosed in FIGS. 22-24), connected to a Network Systemshares a LP or moods with a third party service provider 120, FIG. 1(b),that allocates a resource 22, including but not limited to a CPU andmemory, to process the request and to store the LP and associatedinformation into a Data Store 146 (FIG. 1(e).

124, the cloud or mobile device CPU and memory to process and associatedinformation in the data store.

A client device 124, FIG. 1(b), exchanges messages with the InterfaceComponent 126, FIG. 1(b), according to communication protocols with athird party service provider 120, FIG. 1(b), that allocates a resource122, FIG. 1(b), including but not limited to a CPU and memory, toprocess the message, to acknowledge it, respond to it and to store a logof this message into a Data Store 146, FIG. 1(e).

FIG. 17 illustrates 3DoM general architecture 1700. Architecture 1700comprises 3DoM® cloud or mobile device clients 1710, 3DoM® cloud ormobile device server 1720, and a back end infrastructure 1730. The 3DoM®cloud or mobile device clients 1710 are represented in a layercontaining all 3DoM® clients, including Network System web browsers ondesktops, mobile devices, tablets, and the like. The back endinfrastructure 1730 supports the entire architecture but, in particular,data storage servers.

FIG. 18 illustrates the 3DoM® logical architecture 1800. Front end 1810represents the main entrance door to the 3DoM® Platform, and it tunnelsmessages coming into the system from users 1820 routing them to the3DoM® Platform. It is responsible of handling secure connections viaHTTPS via firewall 1830. An information bus 1840 represents thecollection of technologies used to replicate data and make the 3DoM®Platform scalable and it is responsible for load balancing and datareplication as well as scalability. Connected to the information bus1840 are a number of 3DoM® servers 1850, 1860, 1870 and two databaseservers 1880, 1890. Each 3DoM® server 1850, 1860, 1870 implements the3DoM® Platform and each database server 880, 1890 stores data.

The three-dimensional model may be changed locally on a mobile device;it may also be stored locally on the mobile device.

In one embodiment, the three-dimensional model is stored on a platformin a cloud environment. From this platform, the three-dimensional modelcan be shared via the platform with at least one social network.Optionally, the three-dimensional model is integrated with the at leastone network. This means that a user, having created his/herthree-dimensional model, can share it with his/her contacts on his/hersocial networks. Additionally, his/her contacts may also havethree-dimensional models which are shared on their social networks viathe platform.

In addition, in one embodiment, emotion of a user can be shared on atleast one social network via the platform using the three-dimensionalmodel. The three-dimensional model of a sender of a communication may beanimated according to his/her feelings and/or mood. In this way, arecipient of the communication can be made aware of the feelings and/ormood of the sender as he/she receives the communication.

Virtual objects may be sent to a recipient via the platform connected tothe at least one social network. A recipient, on receipt of the virtualobject, may update his/her three-dimensional model in accordance withthe virtual object to display, in response to the type of virtualobject, one or more of: simultaneous facial emotions; successive facialemotions; animations; graphical object displays; and text messagedisplays. Such an update is achieved in accordance with a storyboard asdescribed in more detail below.

In one embodiment, the virtual object can be selected from a pluralityof virtual objects made available on the platform.

Application programming interfaces (APIs) are provided on the platformthrough which of the following functions can be performed:

-   -   the three-dimensional model can be animated, for example, by the        creation of a succession of animations, in the form of        storyboards;    -   a user is targeted using advertising related to his/her mood.        Although two examples of API are given here, other API        applications are also possible.

The ability to access three-dimensional models is provided by an APIwhich allows any third party Network System and mobile deviceapplication to replace two-dimensional images with three-dimensionalmodels created in accordance with one embodiment of the presentinvention.

By incorporating the technology underlying the present invention intotheir applications, including but not limited to Network System, mobiledevice and the like. Application developers can offer a more engagingand emotional user experience thereby resulting in more faithful andsatisfied users.

In one embodiment, three-dimensional models can be customized usingaccessories worn and/or displayed on and/or around a face. Packages ofpersonalized accessories can be created and made available on theplatform. In addition, packages of personalized moods may also becreated and made available on the platform.

The three-dimensional models created in accordance with one embodimentthe present invention can be used to substitute for two-dimensionalimages in Network System and mobile device applications. In oneembodiment, the three-dimensional models can be utilized in videogamesor other applications using models of characters.

Moreover, in some embodiments, the present invention allows for thecreation of an image wall including a plurality of facial images formingpart of a patchwork, each image comprising a screenshot of athree-dimensional model.

As a non-limiting example, in one embodiment of the present invention,one or more applications and an application programming interface (API)are provided The API can be an app at a mobile device as disclosedhereafter.

As non-limiting examples, mobile devices, desktop platforms and the likecan be enhanced to implement a wide range of so-called “hi-tech”applications including: interactive three-dimensional (3D) games andvirtual worlds; humanoid robotics and artificial intelligence (AI)interaction; performance-driven avatar animation; smart screens,billboards and kiosks; M-commerce; 3D displays; advanced and adaptivevisualizations; and intelligent video conferencing.

With reference to FIG. 2, a system 130 includes the third party serviceprovider 120 that supports any number of resources 122 (e.g., hardware,software, firmware, and the like) that can be employed by the clientdevice 124 and/or disparate client device(s) not shown. The third partyservice provider 120 further comprises the interface component 128 thatreceives resource utilization requests, including but not limited torequests to effectuate operations utilizing resources 122 supported bythe third party service provider 120 from the client device 124 and thedynamic allocation component 110 that partitions resources 122,including but not limited to, between users, devices, computationaltasks, and the like. Moreover, the dynamic allocation component 110 canfurther include a user state evaluator 132, an enhancement component 134and an auction component 136.

The user state evaluator 132 can determine a state associated with auser and/or the client device 124 employed by the user, where the statecan relate to a set of properties. For instance, the user stateevaluator 132 can analyze explicit and/or implicit information obtainedfrom the client device 124 (e.g., via the interface component 128)and/or retrieved from memory associated with the third party serviceprovider 120 (e.g., preferences indicated in subscription data). Staterelated data yielded by the user state evaluator 132 can be utilized bythe dynamic allocation component 110 to tailor the apportionment ofresources 122.

According to another illustration, the user state evaluator 132 canconsider characteristics of the client device 124, which can be used toapportion resources 122 by the dynamic allocation component 110. Forinstance, the user state evaluator 132 can identify that the clientdevice 124 is a mobile device with limited display area. Thus, thedynamic allocation component 110 can employ this information to reduceresources 122 utilized to render an image upon the client device 124since the cellular telephone may be unable to display a rich graphicaluser interface.

Moreover, the enhancement component 134 canfacilitate increasing anallocation of resources 122 for a particular user and/or client device124.

Referring to FIG. 3, a system 138 can be utilized that employs loadbalancing to optimize utilization of resources 122. The system 138includes the third party service provider 120 that communicates with theclient device 124 (and/or any disparate client device(s) and/ordisparate third party service provider(s)). The third party serviceprovider 120 can include the interface component 128 that transmitsand/or receives data from the client device 124 and the dynamicallocation component 110 that allots resources 122. The dynamicallocation component 110 can further comprise a load balancing component140 that optimizes utilization of resources 122.

Moreover, the load balancing component 140 can monitor resources 122 ofthe third party service provider 120 to detect failures. If a subset ofthe resources 122 fails, the load balancing component 140 can continueto optimize the remaining resources 122. Thus, if a portion of the totalnumber of processors fails, the load balancing component 140 can enableredistributing cycles associated with the non-failing processors.

Now turning to FIG. 4, a system 142 is illustrated that archives and/oranalyzes data utilizing the third party service provider 120. The thirdparty service provider 120 can include the interface component 128 thatenables communicating with the client device 124. Further, the thirdparty service provider 120 comprises the dynamic allocation component110 that can apportion data retention resources, for example. Moreover,the third party service provider 120 can include an archive component144 and any number of data store(s) 146. Access to and/or utilization ofthe archive component 144 and/or the data store(s) 146 by the clientdevice 124 (and/or any disparate client device(s)) can be controlled bythe dynamic allocation component 110. The data store(s) 146 can becentrally located and/or positioned at differing geographic locations.Further, the archive component 146 can include a management component148, a versioning component 150, a security component 152, a permissioncomponent 154, an aggregation component 156, and/or a restorationcomponent 158.

The data store(s) 146 can be, for example, either volatile memory ornonvolatile memory, or can include both volatile and nonvolatile memory.By way of illustration, and not limitation, nonvolatile memory caninclude read only memory (ROM), programmable ROM (PROM), electricallyprogrammable ROM (EPROM), electrically erasable programmable ROM(EEPROM), or flash memory. Volatile memory can include random accessmemory (RAM), which acts as external cache memory. By way ofillustration and not limitation, RAM is available in many forms such asstatic RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), doubledata rate SDRAM (DOR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM{SLDRAM), Rambus direct RAM (RDRAM), direct Rambus dynamic RAM (DRDRAM),and Rambus dynamic RAM (RDRAM). The data store(s) 146 of the subjectsystems and methods is intended to comprise, without being limited to,these and any other suitable types of memory. In addition, it is to beappreciated that the data store(s) 146 can be a server, a database, ahard drive, and the like.

The management component 148 facilitates administering data retained inthe data store(s) 146. The management component 148 can enable providingmulti-tiered storage within the data store(s) 146, for example.According to this example, unused data can be aged-out to slower disksand important data used more frequently can be moved to faster disks;however, the claimed subject matter is not so limited. Further, themanagement component 148 can be utilized (e.g. by the client device 124)to organize, annotate, and otherwise reference content without making itlocal to the client device 124. Pursuant to an illustration, enormousvideo files can be tagged via utilizing a cell phone. Moreover, themanagement component 148 enables the client device 124 to bind metadata,which can be local to the client device 124, to file streams (e.g.,retained in the data store(s) 146); the management component 148 canenforce and maintain these bindings.

Additionally or alternatively, the management component 148 can allowfor sharing data retained in the data store(s) 146 with disparate usersand/or client devices. For example, fine-grained sharing can besupported by the management component 148

The versioning component 150 can enable retaining and/or trackingversions of data. For instance, the versioning component 150 canidentify a latest version of a document (regardless of a saved locationwithin data store(s) 146).

The security component 152 limits availability of resources based onuser identity and/or authorization level. For instance, the securitycomponent 152 can encrypt data transferred to the client device 124and/or decrypt data obtained from the client device 124. Moreover, thesecurity component 152 can certify and/or authenticate data retained bythe archive component 144.

The permission component 154 can enable a user to assign arbitraryaccess permissions to various users, groups of users and/or all users.

Further, the aggregation component 156 assembles and/or analyzescollections of data. The aggregation component 156 can seamlessincorporate third party data into a particular user's data.

The restoration component 158 rolls back data retained by the archivecomponent 144. For example, the restoration component 158 cancontinuously record an environment associated with the third partyservice provider 120. Further, the restoration component 158 canplayback the recording.

A 3D representation of human or animal face, and/or other physicalantimony is called a “Living Portrait®” or LP, which, as its nameindicates, is alive and has the ability to change emotions or have amood of its own. This LP therefore has the ability to become a new layerwhen representing people in the virtual world.

It closes the gap between the real person and the personae that is therepresentation in the virtual world, in other words, the avatar. Thechange in emotions and moods of the LP can be controlled by the user orby a command which acts on the LP through its APL The change in emotionsand moods can be transmitted to his/her friends connected to his/hersocial network.

For a better understanding, FIG. 2 illustrates a “life cycle” 200 of aLP. Using the 3DoM® cloud or mobile device client {which may be a mobiledevice app, a desktop application, a Network System app, a socialnetwork itself as described above, and the like).

As a non-limiting example, a user ‘U’ connects to the 3DoM® Platformusing an existing account selected one from his social network accounts(Facebook®, Twitter®, Google+®, etc.), step 210. “Facebook®”, “Twitter®”and “Google®” are trademarks of Facebook Inc., Twitter Inc., and GoogleInc. respectively. Managing federated authentication, namely, where theuser logs in only once and does not need to remember several logincredentials, through existing social network accounts allows access tothe user's social graph, and the LP application becomes part of thesocial network.

As a non-limiting example, in one embodiment, the user isable to:

1. Create a LP (step 220): using a photo from a photo gallery, a phototaken by a camera in the user's mobile client device 124, FIG. 1(b), ora photo downloaded from the Network System or social network into theuser's mobile, the client device 124, FIG. 1(b). The creation of the LPoccurs either in the client device itself or in a cloud or mobile deviceserver in which case the client device does a pre-processing of thephoto to normalize its file size and then sends it to a third partyservice provider 120, FIG. 1(b), that allocates a resource 122,including but not limited to a CPU and memory, to process the photo andtransform it into a 3D model, the LP. The LP can be stored locally intoa data store 146 (FIG. 1e ), and is sent back to the client devicethrough an interface component 126, FIG. 1(b).

2. Change avatar characteristics (step 230): mood (step 232),expressions (step 234), dress up (step 236), make up (step 238) and thelike. Such changes occur inside the client device 124, FIG. 1(b). If thecharacteristics relate to changes of mood or expressions, the process isdone locally. If the characteristics are dress up or make up, they maybe already stored locally. If not, the Client Device makes a request bysending a message to a third party service provider 120, FIG. 1(b), thatallocates a resource 122, FIG. 1(b), including but not limited to CPUand memory, to process it, fetches the characteristic from a local DataStore 146, FIG. 1(e), and sends it back to the Client Device through anInterface Component 126, FIG. 1(b).

3. Share the LP through the cloud or mobile device (step 240) where theclient device 124, FIG. 1(b), sends a message containing the LP to athird party service provider 120, FIG. 1(b), that allocates a resource122, FIG. 1(b), including but not limited to CPU and memory, to processit, stores the LP in a local Data Store 146 (FIG. 1(e)) and notifiesother client devices with whom the LP is to be shared through anInterface Component 126, FIG. 1(b).

In one embodiment, a user may also access Platform services such as LPcreation without previously authenticating, in which case the LP willnot be associated with a social network account.

In one embodiment, every time the user ‘U’ creates or modifies his/herLP, it is immediately shared through the 3DoM® Platform hosted in thecloud (sometimes referred to as cloud or mobile device server forsimplicity): the user's client device 124, FIG. 1(b), sends a messagecontaining the updates to his LP to a third party service provider 12,FIG. 1(b), that allocates a resource 122, FIG. 1(b), including but notlimited to a CPU and memory, to process it and stores the update to theLP in a local Data Store 146 (FIG. 1(e).

All the connected cloud or mobile device clients, that is, friends ofthe user, receive notification about the changes and they can update theLP on their device from the 3DoM® Platform (step 250): the Third PartyService Provider 120, FIG. 1(b), notifies the client devices of thefriends of that user through an Interface Component 126, FIG. 1(b). Allusers connected with the LP are immediately notified of a change of moodand dressing up of the LP (step 250). The information about connectionsbetween such users is defined by the social graph provided by theconnected social network or by other connection mechanisms provideddirectly by the 3DoM Platform and this information is stored in a localdata store 146 (FIG. 1(e)) of the third party service provider 120, FIG.1(b), which allocates resource 122, FIG. 1(b), including but not limitedto a CPU and memory, to process it and update it periodically.

As part of the connection of a client device 124, FIG. 1(b), to the3DoM® Platform hosted on third party provide 120, FIG. 1(b), (step 210),a notification is sent by the third party service provider 120, FIG.1(b), which allocates a resource 122, FIG. 1(b), including but notlimited to a CPU and memory, to process it and notify the client devicesthrough an Interface Component 126, FIG. 1(b). Upon receiving thenotification, (step 260) the client device reloads the LP (step 270).The reloaded LP is stored locally (step 280) (on the client device).Similarly, when the LP is created for the first time in step 220, it isstored locally as shown in step 280.

Even if this is a simple scenario, the possible applications are manyand some of these applications will be described in more detail below.

A 3DoM® cloud or mobile device client is represented by an applicationon each kind of mobile device that is connected to the 3DoM® cloud ormobile device server. A client is an application with some features thatcan vary in numbers and characteristics depending on the target for theclient itself. For example, a Facebook client may show only the LP onthe Facebook user page (instead of the picture of the profile). Thiskind of client does not have any other features than the visualizationof an LP given as parameter.

To access the V services, the client device 124 connects, in the generalcase, to the 3DoM® Platform hosted on third party provider 120, FIG.1(b), (step 210), which allocates a resource 122 FIG. 1(b), includingbut not limited to a CPU and memory, to process it and supply therequested services and information back to the client device through anInterface Component 126, FIG. 1(b).

To navigate through these activities, the client device 124, FIG. 1(b),may store locally only part (in one embodiment, the most recentinformation limited to a set number of days) of the information aboutthese activities while the complete information is stored on the 3DoMPlatform. In the general case, the client device 124, FIG. 1(b), willtherefore connect to the 3DoM® Platform hosted on third party provider120, FIG. 1(b), (step 210), which allocates a resource 122, FIG. 1(b),including but not limited to a CPU and memory, to process it and supplythe requested information back to the client device through an InterfaceComponent 126, FIG. 1(b).

In the general case, the client device 124, FIG. 1(b), will connect andrequest these information to the 3DoM® Platform hosted on third partyprovider 120, FIG. 1(b), (step 210), which allocates a resource 122,FIG. 1(b), including but not limited to a CPU and memory, to process itand supply the requested information back to the client device throughan Interface Component 126, FIG. 1(b). The user will then be able toread and modify this information locally on the client device 124, FIG.1(b), and to save the new settings by connecting again to the 3DoM®Platform hosted on third party provider 120, FIG. 1(b), (step 210),which allocates a resource 122, including but not limited to a CPU andmemory, to process it and store the new information in a local DataStore 146 (FIG. 1(e). In the general case, these activities take placeon the client device 124, FIG. 1(b), without a need to connect to the3DoM® Platform nor to Network Systems. The client device allocates localresource, including but not limited to a CPU and memory, to processthem.

accessories like hats, wigs, glasses or other kind of gadgets. Inspecific cases, the accessories may already be stored locally on theuser client device. If this is not the case, the Client Device 124, FIG.1(b), makes a request by sending a message to the 3DoM Platform storedon a third party service provider 120, FIG. 1(b), that allocates aresource 122, FIG. 1(b), including but not limited to CPU and memory, toprocess it. The 3DoM Platform identifies that this message relates tothe 3DoM shop which stores all Accessories and dispatches it as such tothe shop. The shop fetches the Accessory from a local Data Store 146,FIG. 1(e), and sends it back to the 3DoM Platform which sends it back tothe Client Device through an Interface Component 126, FIG. 1(b). Suchexchange is logged by the 3DoM Platform as previously described.

In sub-module 434, the user can use expressions to create differentmoods on his/her LP. In general, the instructions to animate the LP withthe expressions will already be stored locally on the user clientdevice. When the use plays on his Client Device 124, FIG. 1(b), andchanges the expressions of his LP, no communication take place with the3DoM Platform stored on a third party service provider 120, FIG. 1(b).In one embodiment, however, if sufficient data bandwidth and/orprocessing power is available at a reasonable cost to the user,expressions may then be uploaded to the 3DoM® Platform for sharingthrough the cloud or mobile device. The mood is uploaded by the user'sclient device 124, FIG. 1(b), that connects and sends a messagerepresenting the mood update to the 3DoM® Platform hosted on third partyprovider 120, FIG. 1b , (step 210), which allocates a resource 122,including but not limited to a CPU and memory, to process it, to log itwith the Archive Component 144 (FIG. 1e ). The 3DoM Platform notifiesthe user's client devices through an Interface Component 126, FIG. 1(b).If the users to be notified are not available (for instance thesmartphone is switched off or not connected to a Network System), theywill be notified the next time they connect to the 3DoM Platform. In oneembodiment, some moods are simple, for example, displaying happiness,tiredness, boredom, angriness etc. or more complex, for example, “I'm apirate” and “I feel like a zombie”. All the client devices are preloadedwith a default set of moods. The user also can create his/her own moodsand upload them to the cloud or mobile device. This is described in moredetail below.

In one embodiment the Store or other stores are associated with themobile device. In the general case, the client device 124, FIG. 1(b),will store locally some information which is the most recent while othermost information is stored on the 3DoM Platform or other devices andsystems connected through Network Systems. The Client Device willconnect to the 3DoM® Platform hosted on third party provider 120, FIG. 1(b), (step 210), which allocates a resource 122, including but notlimited to a CPU and memory, to process it and an exchange ofinformation will occur through an Interface Component 126, FIG. 1(b),whereby the most recent information will be transferred to the userclient device. Mechanism such as time stamp or equivalent may be used inone embodiment.

Credits can be bought using real money (step 610), in which case, theuser, with his client device 124, FIG. 1(b), will connect via a NetworkSystem to a payment service and make a payment by providing a creditcard, PayPal account details or will do an In App Purchase that will beprocessed by the App Store to which the client device is connected.Credits can also In the general case, the client device 124, FIG. 1(b),will connect and request information about which activities give howmany rewards to the 3DoM® Platform hosted on third party provider 120,FIG. 1b , (step 210), which allocates a resource 122, including but notlimited to a CPU and memory, to process it and supply the requestedinformation back to the client device through an Interface Component126, FIG. 1(b). When the user conducts such activities on his clientdevice, such activities will be validated by a connection to the 3DoM®Platform hosted on third party provider 120, FIG. 1b , (step 210), whichallocates a resource 122, including but not limited to a CPU and memory,to process it, to calculate the new reward status of the user and tosend new information back through an Interface Component 126, FIG. 1(b).

The client device 124, FIG. 1(b), will connect and request theseinformation to the 3DoM® Platform hosted on third party provider 120,FIG. 1b , (step 210), which allocates a resource 122, including but notlimited to a CPU and memory, to process it and supply the requestedinformation back to the client device through an Interface Component126, FIG. 1(b). To conduct this activity, the user will create suchgadgets with own tools and/or tools provided by the 3DoM Platform (on oroffline). The user client device 124, FIG. 1(b), will connect and submitthe proposed gadgets to the 3DoM® Platform hosted on third partyprovider 120, FIG. 1b , (step 210), which allocates a resource 122,including but not limited to a CPU and memory, to process and submitthem to a validation process operated by MoodMe. Upon acceptance, theuser will be notified through an Interface Component 126, FIG. 1(b),that his gadget(s) are now being offered for sales on the 3DoM shop andhis gadget(s) will be stored in a local Data Store 146 (FIG. 1(e)) alongwith associated information such as commercial (price, whether in realor virtual currency, country where the gadget is available,characteristics of the gadget, etc.).

Upon rejection, the user will be notified through an Interface Component126 FIG. 1(b), possibly providing the user with motivation andindication of modifications to implement so that his gadget(s) can beoffered for distribution on the 3DoM shop. To do this, the client device124, FIG. 1(b), sends a message containing the 3me to the 3DoM platformhosted by third party seNice provider 120, FIG. 1(b), who allocates aresource 122, FIG. 1(b), including but not limited to a CPU and memory,including but not limited to a CPU and memory, to process it. Thesending of the 3me is then logged in a log file managed by the archivecomponent 144 (FIG. 1e ) and sends the 3me to the recipient through aninterface component 126, FIG. 1 (b). To request a friend's LP, theuser's client device 124, FIG. 1(b), will connect and send a message tothe 3DoM® Platform hosted on third party provider 120, FIG. 1(b), whichallocates a resource 122, FIG. 1(b), including but not limited to a CPUand memory, to process it and to send a message to the user's friendrequesting to share his/her LP through an Interface Component 126, FIG.1(b). The user receiving the request may accept it or not. His clientdevice 124, FIG. 1(b), will send a message back to the 3DoM Platform. Incase of acceptance, his LP will be sent back to the user client devicewho initiated the request either directly from client device to clientdevice if both devices are simultaneously available or through the 3DoMPlatform which may store the LP in a local Data Store 146 (FIG. 1(e).

In case of refusal, a message will be sent by the user client devicerefusing to share the LP back to the 3DoM Platform which will notifyback the user client device who initiated the request. The informationabout this request and its outcome will be logged as described, usingthe Archive Component.

3me® (or MoodEe®) objects may fall within at least To perform this, theuser's client device 124, FIG. 1(b), connects and sends a messagerepresenting the request to update his profile picture on a given socialnetwork to the 3DoM® Platform hosted on third party provider 120, FIG.1b , (step 210), which allocates a resource 122, including but notlimited to a CPU and memory, to process it. The 3DoM Platform connectsto the Social Network on behalf of the user through that social networkAPI and an Interface Component 126, FIG. 1(b), to perform the request.The 3DoM Platform notifies the user about the success. It can also beused for a request through an Interface Component 126, FIG. 1(b), andlogs it with the Archive Component 144 (FIG. 1e ).

This preceding follows similar steps involving the user client device,the 3DoM Platform and the Interface Component.

In one embodiment cloud components including but not limited to a CPUand memory. 126, FIG. 1(b), are used to request information about Userswho had a specific Mood at a very recent moment or other characteristicsas specified by the Content Provider. These requests are processed byallocating resources 122, including but not limited to a CPU and memory126. The requests are logged in the Archive Component 144 (FIG. 1(e))and the requested information is sent back to the Content Provider.Content Providers may submit content (advertising or other) to MoodMeusers who match the Mood state or other characteristics through the APIand the Interface Component 126, FIG. 1(b). The 3DoM Platform willretarget the content to the users' client device 124, FIG. 1(b), throughthe Interface Component 126, FIG. 1(b). It will also log this in theArchive Component 144 (FIG. 1(e)) and will compute billing elements tocharge the Content Provider for the service.

The 3DoM® platform has the knowledge of its users' social circles.Because of this, 3DoM® allows the combination of the LP of a user withthose of his/her friends in his social network and enables thedevelopment of games with a more personal, attractive and engagingexperience: each user plays with his/her LP as the hero and the LPs ofhis/her friends as opponents or team members.

As described above, the creation of a LP is core to the 3DoM® Platform,and requires the ability to transform a two-dimensional (2D) human oranimal portrait into a 3D deformable model.

In one embodiment, the present invention can be implemented as apersonal assistant to create a user-friendly original alerting and/orassistance tool as shown in FIGS. 19 to 21. As a non-limiting example inFIGS. 19 to 21 the personal assistant can be configured to notify theuser: of the name of an incoming caller; of the availability of newapps; news; events; weather; and that the battery is too low, and thelike The personal assistant can also be used with incoming messages(SMS) and email. The personal assistant can be created as describedabove by mapping a chosen face over an existing mesh, including but notlimited to a polygonal model, a mesh model and the like, without theneed for a dedicated server to process images with all the processingbeing done on the mobile device.

3D avatars can be integrated with the user's mobile device contact list.Such avatars can automatically be created by tracking a suitable pictureassociated with a contact name, either taken by the camera on the mobiledevice or one downloaded from a social network, and turning the pictureinto a realistic and animated 3D model as described above.

It will be appreciated that the present invention is not limited tosocial applications and the LP can also be used in other domains, whichcan be as a non-limiting example, e-commerce, in store retail and thelike.

In one embodiment, the present invention can allow people to trymerchandise in a much more engaging and practical way offering benefitssuch as: being able to see themselves with glasses, different make-up,hats etc. It would also be possible to try products or articles notpresent in a store thereby reducing stock and shipping the productdirectly to the user's home. In addition, there is no need to wait for asales person, and it is possible to store favorite times for a finaldecision, getting the opinion of contacts via social networkingconnections. For example, “snapshots” of the LP wearing the item can beposted on the user's social network for feedback of his/her friends.

In addition, a virtual store forming part of the 3DoM® Platform can bemade available to third parties so that these parties can create virtualobjects, such as, 3me® (or MoodEe®), together with associated animationof the LP through an API hosted by the 3DoM® Platform. The virtual storeoffers virtual objects for sale, displays them, groups them by categoryto facilitate selection, and features search functions along with otherfeatures common to e-commerce stores.

Third parties can be offered the possibility of proposing 3me® (orMoodEe®) to the 3DoM® Platform, which, after validation, will bedisplayed in the virtual store, and upon selling these 3me®, willprovide a share of the generated revenue to the third party originator.

The LP provides a high level of realism in terms of facial features andskin texture. The LP can interact with virtual objects received fromfriends to change expression etc.

As non-limiting examples, applications in which the LP in accordancewith the present invention can be used include, but are not limited to:enriching chat or group chat applications, such as, MSN®, SKYPE®,Whatsapp, WeChat, and the like; and to add more emotion and to replaceautomatically the emoticons with a facial expression corresponding tothat emoticon, for example, the :-) emoticon would be replaced by theuser's LP smiling, and likewise for all other emoticons.

In addition, the LP can be used to synchronize lip movement with spokentext to offer hearing impaired users a better interface. Similarly, avirtual reader can be created for reading aloud a Network System page, anews article or any text information using text-to-speech conversiontechnology provided by third parties. In one embodiment, resources 22with a processor and memory that executes software, including but notlimited to algorithms, extract phonemes, basic unit of a language'sphonology, which is combined with other phonemes to form meaningfulunits, including but not limited to words, morphemes and the like, andto animate the mouth of the model. The list of phonemes can be used toanimate the avatar lips. In one embodiment, each phoneme has a specificposition of the mouth and lips or other body component

The virtual reader could be customized to speak with a given accent andvoice, the lip movements being synchronized with the spoken text withthe LP showing expressions and animations to make the reading morelifelike.

In one embodiment, electronic greetings cards could also be createdusing an LP, and, by attaching a message to it, a virtual or a recordedvoice could present the message of the sender while showing facialanimations.

As non-limiting examples, LPs can be incorporated into organizerapplications where the application reminds the user that he/she needs todo a task. If he/she does not do the task, the LP could take adisappointed or angry look, for example. If the task is completed, theLP could show a congratulating face. Likewise, the LP can be used onhigh end gym equipment to give positive feedback when the user isperforming well, for example, better than last he/she did the sameexercise, and to provide encouragement if he/she is struggling with theexercise. Naturally, the LP could also give negative feedback ifnecessary.

In one embodiment, the 3DoM® platform allows the integration with socialnetworks through their AP! and with Network System sites as describedabove. One possible application of the 3DoM® platform is to provide thefunctionality of creating a large image wall composed of a large numberof small facial images, each providing a piece of a sort of patchwork.The image wall is in effect composed of small (thumbnail) images whichare screenshots of LPs.

In one embodiment, the user can put his/her face on the image wall bypurchasing a special type of 3me® (or MoodEe®) which will offer thefollowing possibilities: take a screenshot of his/her LP as it was atits last update; give the user the possibility of inserting a textmessage; and upload the screenshot to the image wall. The image wall canbe implemented either within the social network making use of the APIsof that social network, on the 3DoM® platform or in a Network Systemsite. A user visualizing the image wall would see initially a collectionof static images of the faces. When zooming closer, and/or when doing amouse over (if he/she is using a desktop PC) or a touch (using atablet), the faces would regain their “life” and turn back to theiroriginal Living Portrait format

As a non-limiting example one way to implement the construction of theimage wall is for charitable purposes. Initially, the wall comprises aplurality of grey images on a white background. By sending a specific3me® (or MoodEe®), the face of the user is posted in one of the greyimages which become colored showing that a contribution to thecharitable cause has been made at some point. A user can repeatedly sendsuch 3me® (or MoodEe®) to have his/her picture posted several times.When a user looks at the wall from far, he/she can visualize groups ofcolored pixels in various spots, which correspond to where a user hassent a 3me® (or MoodEe®). The more users send 3me® (or MoodEe®), ineffect, in this instance, making a donation, the more colored groups ofpixels start replacing grey points of the image wall.

Although the image wall has been described with respect to donating tocharity, it could be used for other collections and populated by thesending of specific 3me® (or MoodEe®).

As described above, 3me® (or MoodEe®) are complex objects which produceeffect that are applied to LPs. However, some 3me® (or MoodEe®) objectsmay degrade after a predetermined period of time if not utilized. Usershaving unused 3me® (or MoodEe®) objects will be notified if theseobjects are about to suffer degradation. This gives the user theopportunity to use the object thereby reactivating them for a furtherpredetermined period of time. A user has the option of opting in or outfor such notifications.

Further uses of LPs may include the setting of a mode such as “do notdisturb” mood or similar. When a call is placed to a user who hasselected the “do not disturb” mood, the LP of that user with a facialexpression corresponding to this mood will appear. In addition, amessage may also be set by the user to indicate more details, forexample, “in a meeting” or “sorry I can't take your call”. For this tobe implemented, the caller needs to check with the 3DoM® platform toobtain the user mood information.

In one embodiment, the 3DoM® platform allows users to specify their moodassociated with a communication, and may store the mood of userstogether with the history of mood changes. This information can beanalyzed to provide crowd sentiment analysis.

In one embodiment, LPs may also be enriched with characteristics whichare “Tamagotchi®-like” (Tamagotchi® is a handheld digital pet,Tamagotchi® is a trademark of Bandai Co. Ltd., a subsidiary of NamcoBandai of Taito, Tokyo, Japan.). Users will need to conduct activities,for example, feeding, playing, accessorizing, etc., to keep their LP andtheir accessories in good shape and to ensure that he/she grows uphealthy and the accessories stay in good condition to be used. Suchactivities can be obtained from the 3DoM® platform. The embodiedcharacter in the Tamagotchi® may be a character based on the LP of afriend, a celebrity, a comic book character, etc.

As non-limiting examples, although emotions and moods have beendescribed above as being set by the user, they may be automaticallydetected and uploaded onto the 3DoM® platform. For example, a camera orother device built into a television may capture the emotions and/ormood of one of more persons watching a television show, the captureddata being uploaded to the 3DoM® platform via a Network Systems link.From the 3DoM® platform, this captured information may be used toanimate accordingly the LP of the user. This captured information mayalso be used for sentiment analysis, mood-based commercials and audiencesatisfaction.

In one embodiment, biorhythms of the user may be used. This requires theuse of wearable devices with sensors which can detect changes in heartrate, body temperature etc. to monitor and transmit information whichcan be associated with different emotions.

In one embodiment if a video image of a user is obtained, for example,using a webcam connected to the Network Systems, the emotions of theuser may be detected using facial recognition from the video data. Thisdata can be used to develop an emotional chat by remotely activating anLP showing the emotion detected corresponding to the user and displayingit on his/her LP to his/her remote audience. However, if the user doesnot want to be filmed or recorded and would prefer to representhimself/herself as a character, this character would then be animatedaccordingly.

In one embodiment in competitions, debates or other events, a LP can beused to represent the user. Where the results (which can be partial,during the event or final, after the event) can be determined by a votethrough information technology tools, the results can be displayed using3D animated LP with the capability to express human or animal emotions.An automated information system records the votes which can come from avariety of sources, such as social networks and formatted messages. Thevotes are counted and awarded to the different participants to the eventaccording to the rules defined for the event. Each participant to thecompetition is represented graphically by his/her LP. As votes arecollected, the LP is animated with a facial expression which reflectsthe vote count, for example, when victory gets closer, the LP willexpress a happy emotion; when a defeat looms, the LP will express a sador unhappy emotion; if the vote count is tight, the LP can express anervous emotion (with drops of sweat, for instance); and in general, inthe case where are many candidates, the emotion which best correspondsto the situation will be displayed by each LP.

In one embodiment the information tools supporting this representationof the participants to an event as LPs will display the informationvisually through the various media channel available, such as: TVchannels (SD, HD, Ultra HD, Interactive, Simulcast and other format forbroadcast networks); paid or free channels; Network Systems (NetworkSystem site); mobile device applications; desktop applications; andsocial networks etc.

A server, which can be in the cloud or mobile device as described aboveor an equivalent structure in a mobile device as described hereafter,controlling the event may interact with social networks, RSS feeds,databases, as well as, video information, information received fromcomputers, mobile devices and the Network System.

As a non-limiting example, an extension of this event scenario is toprovide a mobile device application which allows users to vote andbroadcast their emotions on a television show. The television channel(or particular show or program) requests that viewers of the program todownload a suitable application or app from an application store so thatthey can create their own LP, set their mood, vote for one of theparticipants, and, optionally add a short text message. Each viewer'svote is sent either directly to the 3DoM® platform or to 3DoM® enabledprocessor and memory that executes software, including but not limitedto algorithms, made available to the television channel or show. Thevote may be free or may be a paid purchase. In this latter case, theprice can be fixed or variable, and can be set by a real-time auction.

In one embodiment the television program will broadcast a selection ofthe votes in a manner such as a stock market, for example, like a tickertape scrolling horizontally at the bottom of the television screenoverlaying the program image. The ticker tape may include the LP of thevoter showing a living facial expression expressing a mood set by thevoter, a short text message, and the name of the voter. The selectionmay be random, for example, the voter being chosen like in a lottery,based on the price paid by the viewer, or based on any other businessrule set by the 3DoM® platform. Viewers may be charged only in the eventthat their LP is broadcast or may be charged regardless of whether therelevant LP has been broadcasted. In the latter case, the charge may befor use of the 3DoM® platform for viewers not having a 3DoM® platformID.

In one embodiment the 3DoM® platform will enable the followingadditional services: an in application notification informing a viewerthat his/her vote has been accepted and that they will be broadcast livein a few minutes; users will in turn be able to share this event withtheir friends via their social networks with a simple operation in a fewclicks, passing the link of the television show or program if it isbeing broadcast on the Network System or another link or informationallowing people to tune in and watch the show; and selling the footageof a short video sequence containing the broadcast of the LP ticker tapeat the bottom of the screen.

The present invention is not limited to human or animals and it may bepossible to create LPs of other creatures including imaginary ones. SuchLPs can be used in “online competitions” or in interactive games. Inother embodiments, a mobile device is used in place of the cloud. Themobile device can include an app for the methods disclosed herein.

Referring to FIGS. 22-24, the mobile or computing device can include anapp for executing the methods of the present invention. Atouch-sensitive display can be used, sometimes called a “touch screen”for convenience, and may also be known as or called a touch-sensitivedisplay system. The mobile or computing device may include a memory(which may include one or more computer readable storage mediums), amemory controller, one or more processing units (CPU's), a peripheralsinterface, Network Systems circuitry, including but not limited to RFcircuitry, audio circuitry, a speaker, a microphone, an input/output(1/0) subsystem, other input or control devices, and an external port.The mobile or computing device may include one or more optical sensors.These components may communicate over one or more communication buses orsignal lines.

It should be appreciated that the mobile or computing device is only oneexample of a portable multifunction mobile or computing device, and thatthe mobile or computing device may have more or fewer components thanshown, may combine two or more components, or a may have a differentconfiguration or arrangement of the components. The various componentsshown in, FIG. 21 may be implemented in hardware, software or acombination of hardware and software, including one or more signalprocessing and/or application specific integrated circuits.

Memory may include high-speed random access memory and may also includenon-volatile memory, such as one or more magnetic disk storage devices,flash memory devices, or other non-volatile solid-state memory devices.Access to memory by other components of the mobile or computing device,such as the CPU and the peripherals interface, may be controlled by thememory controller.

The peripherals interface couples the input and output peripherals ofthe device to the CPU and memory. The one or more processors run orexecute various software programs and/or sets of instructions stored inmemory to perform various functions for the mobile or computing deviceand to process data.

In some embodiments, the peripherals interface, the CPU, and the memorycontroller may be implemented on a single chip, such as a chip. In someother embodiments, they may be implemented on separate chips.

The Network System circuitry receives and sends signals, including butnot limited to RF, also called electromagnetic signals. The NetworkSystem circuitry converts electrical signals to/from electromagneticsignals and communicates with communications networks and othercommunications devices via the electromagnetic signals. The NetworkSystems circuitry may include well-known circuitry for performing thesefunctions, including but not limited to an antenna system, an RFtransceiver, one or more amplifiers, a tuner, one or more oscillators, adigital signal processor, a CODEC chipset, a subscriber identity module(SIM) card, memory, and so forth. The Network Systems circuitry maycommunicate with networks, such as the Internet, also referred to as theWorld Wide Web (WWW), an intranet and/or a wireless network, such as acellular telephone network, a wireless local area network (LAN) and/or ametropolitan area network (MAN), and other devices by wirelesscommunication.

The wireless communication may use any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA), widebandcode division multiple access (W-CDMA), code division multiple access(CDMA), time division multiple access (TOMA), BLUETOOTH®, WirelessFidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and/orIEEE 802.11n), voice over Internet Protocol (VolP), Wi-MAX, a protocolfor email (e.g., Internet message access protocol (IMAP) and/or postoffice protocol (POP)), instant messaging (e.g., extensible messagingand presence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), and/or InstantMessaging and Presence Service (IMPS)), and/or Short Message Service(SMS)), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

The audio circuitry, the speaker, and the microphone provide an audiointerface between a user and the mobile or computing device. The audiocircuitry receives audio data from the peripherals interface, convertsthe audio data to an electrical signal, and transmits the electricalsignal to the speaker. The speaker converts the electrical signal tohuman-audible sound waves. The audio circuitry also receives electricalsignals converted by the microphone from sound waves. The audiocircuitry converts the electrical signal to audio data and transmits theaudio data to the peripherals interface for processing. Audio data maybe retrieved from and/or transmitted to memory and/or the NetworkSystems circuitry by the peripherals interface. In some embodiments, theaudio circuitry also includes a headset jack (FIG. 20). The headset jackprovides an interface between the audio circuitry and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

The 1/0 subsystem couples input/output peripherals on the mobile orcomputing device, such as the touch screen and other input/controldevices, to the peripherals interface. The 1/0 subsystem may include adisplay controller and one or more input controllers for other input orcontrol devices. The one or more input controllers 1 receive/sendelectrical signals from/to other input or control devices. The otherinput/control devices may include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, and joysticks, clickwheels, and so forth. In some alternate embodiments, input controller(s)may be coupled to any (or none) of the following: a keyboard, infraredport, USB port, and a pointer device such as a mouse. The one or morebuttons may include an up/down button for volume control of the speakerand/or the microphone. The one or more buttons may include a pushbutton. A quick press of the push button may disengage a lock of thetouch screen or begin a process that uses gestures on the touch screento unlock the device, as described in U.S. patent application Ser. No.11/322,549, “Unlocking a Device by Performing Gestures on an UnlockImage,” filed Dec. 23, 2005, which is hereby incorporated by referencein its entirety. A longer press of the push button may turn power to themobile or computing device on or off. The user may be able to customizea functionality of one or more of the buttons. The touch screen is usedto implement virtual or soft buttons and one or more soft keyboards.

The touch-sensitive touch screen provides an input interface and anoutput interface between the device and a user. The display controllerreceives and/or sends electrical signals from/to the touch screen. Thetouch screen displays visual output to the user. The visual output mayinclude graphics, text, icons, video, and any combination thereof(collectively termed “graphics”). In some embodiments, some or all ofthe visual output may correspond to user-interface objects, furtherdetails of which are described below.

A touch screen has a touch-sensitive surface, sensor or set of sensorsthat accepts input from the user based on haptic and/or tactile contact.The touch screen and the display controller (along with any associatedmodules and/or sets of instructions in memory) detect contact (and anymovement or breaking of the contact) on the touch screen and convertsthe detected contact into interaction with user-interface objects (e.g.,one or more soft keys, icons, web pages or images) that are displayed onthe touch screen. In an exemplary embodiment, a point of contact betweena touch screen and the user corresponds to a finger of the user.

The touch screen may use LCD (liquid crystal display) technology, or LPD(light emitting polymer display) technology, although other displaytechnologies may be used in other embodiments. The touch screen and thedisplay controller may detect contact and any movement or breakingthereof using any of a plurality of touch sensing technologies now knownor later developed, including but not limited to capacitive, resistive,infrared, and surface acoustic wave technologies, as well as otherproximity sensor arrays or other elements for determining one or morepoints of contact with a touch screen.

A touch-sensitive display in some embodiments of the touch screen may beanalogous to the multi-touch sensitive tablets described in thefollowing U.S. Pat. No. 6,323,846 (Westerman et al.), U.S. Pat. No.6,570,557 (Westerman et al.), and/or U.S. Pat. No. 6,677,932(Westerman), and/or U.S. Patent Publication 2002/0015024A1, each ofwhich is hereby incorporated by reference in their entirety. However, atouch screen displays visual output from the portable mobile orcomputing device, whereas touch sensitive tablets do not provide visualoutput.

A touch-sensitive display in some embodiments of the touch screen may beas described in the following applications: (1) U.S. patent applicationSer. No. 11/381,313, “Multipoint Touch Surface Controller,” filed May12, 2006; (2) U.S. patent application Ser. No. 10/840,862, “MultipointTouchscreen,” filed May 6, 2004; (3) U.S. patent application Ser. No.10/903,964, “Gestures For Touch Sensitive Input Devices,” filed Jul. 30,2004; (4) U.S. patent application Ser. No. 11/048,264, “Gestures ForTouch Sensitive Input Devices,” filed Jan. 31, 2005; (5) U.S. patentapplication Ser. No. 11/038,590, “Mode-Based Graphical User InterfacesFor Touch Sensitive Input Devices,” filed Jan. 18, 2005; (6) U.S. patentapplication Ser. No. 11/228,758, “Virtual Input Device Placement On ATouch Screen User Interface,” filed Sep. 16, 2005; (7) U.S. patentapplication Ser. No. 11/228,700, “Operation Of A Computer With A TouchScreen Interface,” filed Sep. 16, 2005; (8) U.S. patent application Ser.No. 11/228,737, “Activating Virtual Keys Of A Touch-Screen VirtualKeyboard,” filed Sep. 16, 2005; and (9) U.S. patent application Ser. No.11/367,749, “Multi-Functional Hand-Held Device,” filed Mar. 3, 2006. Allof these applications are incorporated by reference herein in theirentirety.

The touch screen may have a resolution in excess of 1000 dpi. In anexemplary embodiment, the touch screen has a resolution of approximately1060 dpi. The user may make contact with the touch screen using anysuitable object or appendage, such as a stylus, a finger, and so forth.In some embodiments, the user interface is designed to work primarilywith finger-based contacts and gestures, which are much less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, the mobile orcomputing device may include a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad may be a touch-sensitive surfacethat is separate from the touch screen or an extension of thetouch-sensitive surface formed by the touch screen.

In some embodiments, the mobile or computing device may include aphysical or virtual click wheel as an input control device. A user maynavigate among and interact with one or more graphical objects(henceforth referred to as icons) displayed in the touch screen byrotating the click wheel or by moving a point of contact with the clickwheel (e.g., where the amount of movement of the point of contact ismeasured by its angular displacement with respect to a center point ofthe click wheel). The click wheel may also be used to select one or moreof the displayed icons. For example, the user may press down on at leasta portion of the click wheel or an associated button. User commands andnavigation commands provided by the user via the click wheel may beprocessed by an input controller as well as one or more of the modulesand/or sets of instructions in memory. For a virtual click wheel, theclick wheel and click wheel controller may be part of the touch screenand the display controller, respectively. For a virtual click wheel, theclick wheel may be either an opaque or semitransparent object thatappears and disappears on the touch screen display in response to userinteraction with the device. In some embodiments, a virtual click wheelis displayed on the touch screen of a portable multifunction device andoperated by user contact with the touch screen.

The mobile or computing device also includes a power system for poweringthe various components. The power system may include a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

The mobile or computing device may also include one or more sensors,including not limited to optical sensors. FIG. 30 illustrates how anoptical sensor coupled to an optical sensor controller in 1/0 subsystem.The optical sensor may include charge-coupled device (CCD) orcomplementary metal-oxide semiconductor (CMOS) phototransistors. Theoptical sensor receives light from the environment, projected throughone or more lens, and converts the light to data representing an image.In conjunction with an imaging module 58 (also called a camera module);the optical sensor may capture still images or video. In someembodiments, an optical sensor is located on the back of the mobile orcomputing device, opposite the touch screen display on the front of thedevice, so that the touch screen display may be used as a viewfinder foreither still and/or video image acquisition. In some embodiments, anoptical sensor is located on the front of the device so that the user'simage may be obtained for videoconferencing while the user views theother video conference participants on the touch screen display. In someembodiments, the position of the optical sensor can be changed by theuser (e.g., by rotating the lens and the sensor in the device housing)so that a single optical sensor may be used along with the touch screendisplay for both video conferencing and still and/or video imageacquisition.

The mobile or computing device may also include one or more proximitysensors. In one embodiment, the proximity sensor is coupled to theperipherals interface. Alternately, the proximity sensor may be coupledto an input controller in the 1/0 subsystem. The proximity sensor mayperform as described in U.S. patent application Ser. No. 11/241,839,“Proximity Detector In Handheld Device,” filed Sep. 30, 2005; Ser. No.11/240,788, “Proximity Detector In Handheld Device,” filed Sep. 30,2005; Ser. No. 13/096,386, “Using Ambient Light Sensor To AugmentProximity Sensor Output”; Ser. No. 13/096,386, “Automated Response ToAnd Sensing Of User Activity In Portable Devices,” filed Oct. 24, 2006;and Ser. No. 11/638,251, “Methods And Systems For AutomaticConfiguration Of Peripherals,” which are hereby incorporated byreference in their entirety. In some embodiments, the proximity sensorturns off and disables the touch screen when the multifunction device isplaced near the user's ear (e.g., when the user is making a phone call).In some embodiments, the proximity sensor keeps the screen off when thedevice is in the user's pocket, purse, or other dark area to preventunnecessary battery drainage when the device is a locked state.

In some embodiments, the software components stored in memory mayinclude an operating system, a communication module (or set ofinstructions), a contact/motion module (or set of instructions), agraphics module (or set of instructions), a text input module (or set ofinstructions), a Global Positioning System (GPS) module (or set ofinstructions), and applications (or set of instructions).

The operating system (e.g., Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS, oran embedded operating system such as VxWorks) includes various softwarecomponents and/or drivers for controlling and managing general systemtasks (e.g., memory management, storage device control, powermanagement, etc.) and facilitates communication between various hardwareand software components.

The communication module facilitates communication with other devicesover one or more external ports and also includes various softwarecomponents for handling data received by the Network Systems circuitryand/or the external port. The external port (e.g., Universal Serial Bus(USB), FIREWIRE, etc.) is adapted for coupling directly to other devicesor indirectly over a network (e.g., the Internet, wireless LAN, etc.).In some embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used on iPod (trademark of Apple Computer, Inc.)devices.

The contact/motion module may detect contact with the touch screen (inconjunction with the display controller) and other touch sensitivedevices (e.g., a touchpad or physical click wheel). The contact/motionmodule includes various software components for performing variousoperations related to detection of contact, such as determining ifcontact has occurred, determining if there is movement of the contactand tracking the movement across the touch screen, and determining ifthe contact has been broken (i.e., if the contact has ceased).Determining movement of the point of contact may include determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations may be applied to single contacts (e.g., onefinger contacts) or to multiple simultaneous contacts (e.g.,“multitouch”/multiple finger contacts). In some embodiments, thecontact/motion module and the display controller also detect contact ona touchpad. In some embodiments, the contact/motion module and thecontroller detects contact on a click wheel.

Examples of other applications that may be stored in memory includeother word processing applications, JAVA-enabled applications,encryption, digital rights management, voice recognition, and voicereplication.

In conjunction with touch screen, display controller, contact module,graphics module, and text input module, a contacts module may be used tomanage an address book or contact list, including: adding name(s) to theaddress book; deleting name(s) from the address book; associatingtelephone number(s), e-mail address(es), physical address(es) or otherinformation with a name; associating an image with a name; categorizingand sorting names; providing telephone numbers or e-mail addresses toinitiate and/or facilitate communications by telephone, videoconference, e-mail, or IM; and so forth.

The foregoing description of various embodiments of the claimed subjectmatter has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit the claimedsubject matter to the precise forms disclosed. Many modifications andvariations will be apparent to the practitioner skilled in the art.Particularly, while the concept “component” is used in the embodimentsof the systems and methods described above, it will be evident that suchconcept can be interchangeably used with equivalent concepts such as,class, method, type, interface, module, object model, and other suitableconcepts. Embodiments were chosen and described in order to bestdescribe the principles of the invention and its practical application,thereby enabling others skilled in the relevant art to understand theclaimed subject matter, the various embodiments and with variousmodifications that are suited to the particular use contemplated.

What is claimed is:
 1. A system comprising: at least one camera capableof acquiring at least one first image comprising a representation of atleast a portion of a face; and at least one processor configured to:detect a plurality of landmarks in the at least one first image, theplurality of landmarks corresponding to respective features of the face;generate a three-dimensional model of the face based, at least in part,on the plurality of detected landmarks; transform the three-dimensionalmodel at least in part in response to video data of the user; andgenerate at least one second image, based at least in part on thetransformed three-dimensional model.
 2. The system of claim 1, whereinthe at least one camera comprises at least one camera integrated in amobile device.
 3. The system of claim 1, wherein the at least one cameracomprises a webcam.
 4. The system of claim 1, wherein the at least oneprocessor is configured to transform the three-dimensional model inresponse to emotion of the user in the video data.
 5. The system ofclaim 1, wherein the at least one processor is configured to apply aplurality of transformations to the three-dimensional model in responseto the video data of the user.
 6. The system of claim 5, wherein the atleast one processor is configured to generate a sequences of images,based at least in part on the plurality of transformations, to producean animation.
 7. The system of claim 6, wherein the animation expressesan emotion based on the video data of the user.
 8. The system of claim7, wherein the animation expresses an emotion corresponding to emotionof the user in the video data.
 9. The system of claim 1, wherein the atleast one processor is configured to transform the three-dimensionalmodel at least in part in response to a request to incorporate at leastone object absent from the at least one first image.
 10. The system ofclaim 9, wherein the at least one object comprises at least oneaccessory, and wherein the at least one processor is configured togenerate at least one second image comprising a representation of atleast a portion of the face and a representation of the at least oneaccessory.
 11. The system of claim 10, wherein the at least oneaccessory comprises an article, a product, a branded item, a hat, a wig,glasses, clothing, a costume and/or a gadget.
 12. The system of claim 9,wherein the at least one object comprises at least one effect applied tothe face, and wherein the at least one processor is configured togenerate at least one second image comprising a representation of atleast a portion of the face having the at least one effect appliedthereto.
 13. The system of claim 12, wherein the at least one effectcomprises a texture and/or coloring applied to the face, and wherein theat least one processor is configured to generate at least one secondimage comprising a representation of at least a portion of the facehaving the texture and/or the coloring applied thereto.
 14. The systemof claim 13, wherein the at least one effect comprises coloring to lipsof the face, and wherein the at least one processor is configured togenerate at least one second image comprising a representation of atleast a portion of the face having the coloring to the lips appliedthereto.
 15. The system of claim 13, wherein the at least one effectcomprises coloring to skin of the face, and wherein the at least oneprocessor is configured to generate at least one second image comprisinga representation of at least a portion of the face having the coloringto the skin applied thereto.
 16. The system of claim 13, wherein the atleast one effect comprises texture to skin of the face, and wherein theat least one processor is configured to generate at least one secondimage comprising a representation of at least a portion of the facehaving the texture to the skin applied thereto.
 17. The system of claim13, wherein the at least one effect comprises make-up, and wherein theat least one processor is configured to generate at least one secondimage comprising a representation of at least a portion of the facehaving the make-up applied thereto.
 18. The system of claim 9, whereinthe request to incorporate the at least one object comprises aselection, by a user, of the at least one object to be incorporated inthe at least one second image.
 19. The system of claim 1, wherein thethree-dimensional model comprises a deformable model and imageinformation from the at least one first image.
 20. The system of claim1, wherein the three-dimensional model comprises a mesh textured withimage information from the at least one first image.
 21. The system ofclaim 1, wherein the at least one processor is configured to transmitthe at least one second image to a remote location.
 22. The system ofclaim 1, wherein the at least one processor is configured to transmitthe at least one second image via a text message.
 23. The system ofclaim 1, wherein the at least one processor is configured to transmitthe at least one second image to a social networking platform.